A Prior-information-guided Residual Diffusion Model for Multi-modal PET Synthesis from MRI

1 Introduction

Alzheimer's disease (AD) is a progressive and irreversible neurodegenerative disorder, and early diagnosis is crucial for effective disease management and improving the patient's life quality [Provost et al., 2021]. Given that AD causes changes in multiple biomarkers, the National Institute on Aging and Alzheimer's Association proposed the joint multibiomarker (amyloid-β, tau, and neurodegeneration) diagnostic approach, called ATN analysis [Jack Jr et al., 2018]. Conducting ATN analysis requires collecting the patient's AβPET, tau-PET, and MRI to acquire information on amyloid-β (Aβ) deposition, tau proteins, and neurodegeneration, respectively. However, the high costs (especially radiation risks) of PET imaging limit the feasibility of conducting multiple PET scans (i.e., Aβ-PET and tau-PET) within a short period, posing challenges to real-time ATN analysis and thus restricting diagnostic accuracy. To achieve cost-effective and realtime ATN analysis, a feasible approach is to simultaneously synthesize corresponding Aβ-PET and tau-PET images from Magnetic Resonance Imaging (MRI).

∗Corresponding author

Synthesizing multi-modal PET images from MRI is not a straightforward task, as it is a one-to-many generative problem, especially when distinguishing similar modalities such as Aβ-PET and tau-PET. The most direct approach for such tasks is to use multiple models for different generative tasks [Lan et al., 2021; Deng et al., 2022]. While simple, this method overlooks the interdependencies inherent in the related generative tasks and incurs computational redundancy due to the need to train multiple models. Consequently, researchers are increasingly leaning towards using a unified model to handle multiple generative tasks simultaneously. One approach in unified model design is to have multiple generative tasks share parts of the model architecture. For instance, [Jiang et al., 2023] employs a shared encoder and two task-specific decoders to simultaneously synthesize two types of dual-energy CT images from single-energy CT images. This method effectively reduces the training burden and captures the dependencies between different generative tasks to some extent. To further enhance the performance of unified models, some studies have explored leveraging text control, enabling different generative tasks through the same model pathway. This strategy significantly reduces model parameters and adequately captures the dependencies between tasks.

Diffusion models are a recently popular type of textcontrolled unified model. They consist of a process that gradually adds noise to an input image until it becomes a pure noise, and also a denoising process that continuously removes noise from the noised samples to recover the input image. The diffusion-based models decompose the generative task into a series of simple steps, allowing for precise control of the generation process using text information. However, they are often guided by simple textual forms, such as "Aβ-PET" and "tau-PET", neglecting some highly relevant prior information associated with the images. Besides, our task involves generating highly similar images (i.e., Aβ-PET and tau-PET) within the same domain from MRI. Traditional diffusion models struggle to differentiate between these two PET modalities effectively, ultimately limiting their performance of generation. Furthermore, despite using the same network to synthesize PET images of different modalities, they do not impose further constraints on the synthesized images of different PET modalities, i.e., their relationships and discrepancy, resulting in suboptimal performance.

To tackle these issues, we propose a novel unified model to simultaneously synthesize multi-modal PET images from MRI. The core of our method is incorporating residual learning into the diffusion model to emphasize inter-domain differences between PET and MRI, rather than their nearly identical anatomical structures. The residual learning not only helps the model differentiate two similar PET modalities but also facilitates a meticulous exploration of distinctive features between the PET domain and the MRI domain. To improve the diagnostic value of synthesized PET images, we take the subject's age and gender as the prior information to guide the diffusion model in synthesizing PET images with semantic consistency. Besides, to constrain the PET images from different modalities, we develop an intra-domain difference loss to ensure that the intra-domain differences among synthesized PET images closely match those among real PET images. The proposed novel components and loss functions can effectively synthesize multi-modal PET images and boost the usability of our model in real-world clinical scenarios.

Our main contributions are summarized as follows:

• Incorporating residual learning into the diffusion model to emphasize inter-domain differences between PET and MRI, thereby forcing each modality to maximally reconstruct its modality-specific details for AD diagnosis.
• Leveraging prior information, such as age and gender, to guide the diffusion model in synthesizing PET images with semantic consistency, enhancing their diagnostic value.
• Developing an intra-domain difference loss to ensure that the intra-domain differences among synthesized PET images closely match those among real PET images, promoting more accurate synthesis.

2 RelatedWorks

2.1 Multi-Model PET Synthesis fromMRI

The high costs of acquiring PET images have limited their clinical use, leading to research efforts aimed at reducing these costs through image generation technologies. Among these efforts, generating corresponding PET images from MRI has garnered significant attention. This research can be divided into three categories based on the modality of the target PET images: 1) Synthesis of FDG-PET fromMRI [Li et al., 2014; Wang et al., 2018; Lan et al., 2021; Pan et al., 2021; Hu et al., 2021]; 2) Synthesis of Aβ-PET from MRI [Kang et al., 2020; Kimura et al., 2020; Jin et al., 2023; Vega et al., 2023]; 3) Synthesis of tau-PET fromMRI [Sun et al., 2022; Lee et al., 2022; Jang et al., 2023]. Current studies mainly focused on synthesizing a single modality of PET images, yet accurate AD diagnosis (ATN analysis) requires multi-modal PET images. Consequently, our objective is to develop a method for synthesizing multi-modal PET images from MRI, offering a rapid and cost-effective solution for precise multibiomarker joint AD diagnosis.

2.2 Residual Learning in Image Generation

Residual learning, initially developed to solve gradient explosion in image recognition tasks [He et al., 2016], focuses on enabling models to learn the differences (i.e., residuals) between inputs and outputs, rather than directly learning the mapping relationship. Due to its effectiveness in making model learning more efficient and targeted, researchers try to apply residual learning to generative tasks as well. For example, [Jifara et al., 2019] used it in autoencoders for the image denoising task, allowing the network to more effectively restore image details. [Gao et al., 2020] proposed a two-level residual learning CNN for image super-resolution tasks to achieve high-resolution images with high-frequency components. [Duan et al., 2021] introduced a residual Generative Adversarial Network (GAN) for the cross-modal translation task to help the model more efficiently learn complex image distributions. [Huang et al., 2023] incorporated the residual learning into the Vision transformer (ViT) to ease the training of deeper ViT and significantly alleviate the degradation problem. Inspired by these works, we incorporate the concept of residual learning into the diffusion process of diffusion models, by proposing a residual diffusion model that can better differentiate similar data distributions.

2.3 Text-Guided Synthesis

Text-guided synthesis is a flexible and effective method for controlling image generation, aimed at producing images that are highly consistent with the associated textual descriptions. Early methods [Choi et al., 2018; Tang et al., 2018; Duan et al., 2021] simply transform input texts into discrete labels, which are then used to control image generation. Although this approach could control image generation to some extent, much of the semantic information from the original text was significantly lost in the process of converting to discrete labels. To address this, researchers have tried to use convolutional network-based feature extraction modules [Rombach et al., 2022] to extract semantic features from texts. However, due to the limited scale of the feature extraction modules, these methods could only handle simple texts and lack flexibility. Fortunately, the development of largescale pre-trained language models [Radford et al., 2021; Nichol et al., 2021; Ramesh et al., 2022] has recently made it possible to extract semantic information from input texts more effectively and flexibly. In this paper, we design a prior-information-guided module based on the Contrastive Language-Image Pre-training model (CLIP [Radford et al., 2021]), which efficiently and flexibly handles various information such as image categories and individual attributes. This module is then used to guide the image generation process.

3 Methodology

The overview of our proposed method is shown in Figure 1 (a). Given an input PET image xi0 (where i = 1 for Aβ-PET, i = 2 for tau-PET), it first undergoes a series of transformations into noisy MRI xiT through the residual diffusion process, gradually merging inter-domain differences. The noisy MRI xiT then undergoes a series of reconstruction steps under the control of the prior-information-guided module until it is restored to the input PET image. We also design an intradomain difference loss to constrain the generation between different PET modalities to improve generation performance.

3.1 Residual DiffusionModel

MRI and corresponding multi-modal PET images (i.e., Aβ and tau-PET) have a substantial proportion of similarities, as they are acquired from the same individual. Using traditional diffusion models to learn the mapping fromMRI to different PET modalities can lead to significant redundancy, and the model might struggle to learn the critical parts (i.e., the differences between MRI and different PET modalities, as shown in Figure 1 (b)). Therefore, inspired by the concept of residual learning, we design a residual diffusion strategy to enable the model to directly learn the differences between MRI and different PET modalities.

In the forward process, we transform the input PET image x0 into a noisy MRI xT through a series of transformations. To achieve this, we first extract the inter-domain difference e0 = y0 - x0 between MRI y0 and the input PET image x0, and then apply a shifting sequence {ηt}Tt=1 to add this difference e0 to the input PET image. Thus, the intermediate version xt can be modeled as:

where η1 → 0, and ηT → 1.

In the reverse process, we progressively restore the noisy MRI xT (i.e., the noisy version of MRI y0) back into the input PET image x0 through a series of reconstruction steps. Representing the posterior distribution as p(x0|y0), this process can be formalized as follows:

where we assume the marginal distributions of xT converges to N(xT ; y0,Σ), i.e., p(xT |y0) ≈ N(xT ; y0,Σ). Following commonly used strategies in diffusion models [Dhariwal and Nichol, 2021; Song et al., 2020], we adopt the assumption of pθ(xt-1|xt, y0) = N(xt-1; µθ(xt, y0, t),Σθ(xt, y0, t)). The Σθ(xt, y0, t) can be viewed as a fixed variance and µθ(xt, y0, t) can be reparameterized as follows:

where fθ is a deep neural network with learnable parameter θ, aiming to predict x0. Consequently, the reconstruction loss of our model is defined as follows:

3.2 Prior-Information-GuidedModule

In this work, we aim to accomplish the generation of both Aβ-PET and tau-PET images by using a single model. To achieve this, we guide each generation with corresponding textual information to produce the expected PET modality. Meanwhile, considering that our generated images are ultimately for diagnostic tasks, we also wish to integrate clinically relevant information, such as age and gender, into the generation process to enhance the diagnostic value of the images. To this end, we utilize a large-scale pre-trained language model, namely CLIP, to design a prior-informationguided module. This module can effectively and flexibly process these textual information to guide image generation.

The details of the prior-information-guided module are provided in Figure 1 (c). Specifically, this module takes a piece of text information as input and then tokenizes the text into concept tokens with a length of 77. The concept tokens are then processed through a text encoder based on ViT-B/32 CLIP to extract a textual vector. The extracted textual vector is subsequently used to guide a series of image reconstruction processes. Noting that considering the original pre-trained CLIP model is designed for natural images and corresponding texts, we employ fine-tuning techniques on the final transformer layer of CLIP to ensure that the model's understanding of the input clinical information can align with the target image generation task.

During each reconstruction step, we guide the reconstruction by fusing the textual vector and the image feature maps extracted from the bottleneck of each reconstruction network, serving as the input for the next layer. We utilize cross-attention to fuse the textual vector and image feature maps, which can be mathematically represented as

where zimage is the flattened image feature maps from the bottleneck layer, and ztext represents the textual vector.

3.3 Intra-Domain Difference Loss

Since Aβ-PET and tau-PET are highly similar, it's quite challenging for the model to accurately reconstruct different modalities of PET images from noisy MRI using only the image category information provided by the prior-informationguided module. Thus, we further design an intra-domain difference loss to constrain the synthesis of PET images across different modalities.

Specifically, in each reconstruction step, we employ the intra-domain difference loss to ensure that the differences between predicted Aβ-PET and tau-PET images at multiple levels align with the differences between the real Aβ-PET and tau-PET images, as shown in Figure 1 (d). This can be formulated as follows:

where ˆx10 and ˆx20 are the synthesized Aβ-PET and tau-PET images at each time step t, and x10 and x20 denote real AβPET and tau-PET images, respectively.

By combining the reconstruction loss (Lrec) and the intradomain difference loss (Lhigh), the comprehensive objective function is formulated as:

where λ is a hyper-parameter to balance different loss terms.

4 Experiments

4.1 Dataset andMetric

To validate our proposed method, we collect 1274 sets of Aβ-PET, tau-PET, and T1-weighted MRI from the public Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, with 1020 sets used for training and 254 sets for testing. Each set of images is acquired from the same individual, along with corresponding AD diagnostic labels, and aligned to a common space by affine registration. To eliminate anisotropy, we resample all images to a size of 128×128×128 with a voxel spacing of 1.5 mm3 . Additional details about the collected dataset can be found in Table 1.

We comprehensively evaluated the experimental results from both the reconstruction and diagnosis aspects. In the reconstruction evaluation, we evaluated the quality of the synthesized images using Mean Absolute Error (MAE), Structural Similarity Index (SSIM), and Peak Signal-to-Noise Ratio (PSNR). As for the diagnosis evaluation, we assess the AD diagnostic value of the synthesized images through the AD diagnosis task using the pre-trained disease diagnosis model in [Pan et al., 2021], and quantify the diagnostic outcomes using Accuracy (ACC), F1 Score (F1S), and the Area Under the Receiver Operating Characteristic Curve (AUC).

4.2 Comparison with State-of-the-Art Methods

We compare our proposed method with five state-of-the-art generative approaches, including: 1) Pix2Pix [Isola et al., 2017], 2) SC-GAN [Lan et al., 2021], 3) TransUNet [Chen et al., 2021], 4) ResViT [Dalmaz et al., 2022], and 5) DDPM [Ho et al., 2020]. In our experiments, Pix2Pix, SC-GAN, and TransUNet use two structurally identical models to separately synthesize Aβ-PET and tau-PET images, whereas ResViT and DDPM employ a single model with control modules to generate different modality PET images. Specifically, ResViT utilizes an availability condition module to randomly mask the target modality for learning, enabling the synthesis of diverse target modalities during the inference stage. DDPM is modified by aligning MRI with noisy PET through channel concatenation and integrating text information (represented as discrete vectors) into the denoising process for multi-model PET synthesis, using a common approach as described in [Xie et al., 2022; Lyu and Wang, 2022; ¨Ozbey et al., 2023]. The quantitative and qualitative results are provided in Table 2 and Figure 2, respectively.

Quantitative Comparison: As shown in Table 2, our proposed method outperforms others, showcasing robust performance in generating both Aβ-PET and tau-PET images. Meanwhile, compared to the traditional diffusion model (i.e., DDPM), we achieve substantial improvements across all metrics. Specifically, for Aβ-PET generation, the improvements in MAE, PSNR, and SSIM are respectively 0.43%, 1.18dB, and 1.44%. For tau-PET generation, the improvements in MAE, PSNR, and SSIM are respectively 0.53%, 0.87dB, and 1.25%. This demonstrates the effectiveness of our targeted improvements to the design of the traditional diffusion model.

Qualitative Comparison: The visual results are provided in Figure 2. From the figure, it is evident that methods based on GAN, including ResViT, TransUNet, SC-GAN, and Pix2Pix, tend to synthesize overly smooth images. In contrast, the PET images generated by our proposed method exhibit the clearest critical details. Moreover, the PET images reconstructed by our approach show the smallest difference from the ground truth, as indicated by the lightest colors in the corresponding error maps. These findings collectively demonstrate that our method achieves superior performance to state-of-the-art methods.

4.3 Ablation Studies

To validate the different components of our proposed method, we use the residual diffusion model as a baseline and selectively incorporate intra-domain difference loss and a priorinformation-guided module to form several variants. The results are provided in Table 3 and Figure 3. As indicated in Table 2, the baseline (1st row) shows improvements upon the addition of either intra-domain difference loss (2nd row) or prior-information-guided module (3rd row), and it achieves the best synthetic results when both are applied simultaneously (4th row). This validates the benefit of our designed intra-domain difference loss and prior-informationguided module in synthesizing PET images. Additionally, the incorporation of the prior-information-guided module resulted in significant improvements in the positive and negative diagnoses of Aβ and tau proteins. This demonstrates that the utilized prior information enhances the synthesis of PET images with greater diagnostic value. Figure 3 illustrates that the synthesized PET images by integrating both intra-domain difference loss and prior-information-guided module matches best with the ground truth, further proving the effectiveness of each proposed component.

5 Discussions

5.1 The Role of Prior Information

To further explore the role of prior information in PET image generation, we design two experimental groups to evaluate the impact of two common types of prior information (namely, gender, and age) on PET image synthesis. For gender information, we create texts with correct gender, no gender, and incorrect gender, and then use these texts to guide the synthesis of PET images. The synthesized PET images are subjected to a gender classification task by using the classification model proposed in [Pan et al., 2021]. The diagnostic results are provided in Figure 4. It can be observed that PET images synthesized with guidance from the correct gender information exhibit the strongest gender classification capability, followed by those synthesized with no gender information and incorrect gender information. This indirectly indicates the sensitivity of image synthesis to the guiding gender information, where correct gender guidance leads to images with greater diagnostic value.

Similar to the assessment with gender information, we evaluate the sensitivity of synthesized images to age information under two scenarios, i.e., with and without age guidance. Specifically, we use a prediction model [Cole et al., 2017] to estimate the actual age based on the generated PET images, with results presented in Figure 5. The results reveal that, under age-guided conditions, whether using generated Aβ-PET or tau-PET images for age prediction, the correlation between predicted and actual ages is significantly higher compared to the non-age-guided scenario. This result further demonstrates that our designed prior-information-guided module effectively enables the model to synthesize PET images highly semantically related to the corresponding textual information.

5.2 Diagnostic Value of Synthetic Images

Furthermore, we evaluate the diagnostic value of the generated images by diagnosing the status of Aβ and tau proteins using the generated Aβ-PET and tau-PET images. The diagnostic results using PET images generated by different methods are presented in Table 4. The statistical results show that diagnoses made using PET images generated by our method achieve results close to those using real images, with accuracies (ACC) of 81.90% and 83.02% for the positive and negative diagnoses of Aβ and tau proteins, respectively. Moreover, compared to other methods, our method achieves the best diagnostic results. These results suggest that our proposed method can synthesize PET images with exceptionally high diagnostic value, showing potential for application in clinical diagnosis.

6 Conclusions

In this paper, we propose a prior-information-guided residual diffusion model to simultaneously generate multi-modal PET images fromMRI, addressing the challenge of acquiring multi-modal PET images within a short time. We specifically design three strategies tailored to this task, including 1) utilizing residual diffusion to enable the model to focus only on the inter-domain differences between MRI and PET, 2) embedding a prior-information-guided module for integrating image modality and subject attributes into the generation process, and 3) designing an intra-domain difference loss to constrain the differences among different PET modalities. Experiments on the ADNI dataset demonstrate that our method outperforms state-of-the-art approaches and shows potential for application in clinical auxiliary diagnosis.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China (grant numbers 62131015, 62250710165), Science and Technology Commission of Shanghai Municipality (STCSM) (grant number 21010502600), STI2030-Major Project (No. 2022ZD0213100), and The Key R&D Program of Guangdong Province, China (grant number 2021B0101420006).