Adaptive Convolutional Dictionary Network for CT Metal Artifact Reduction

1 Introduction

X-ray computed tomography (CT) has been broadly adopted for clinical diagnosis. Nevertheless, common metallic implants within patients, such as dental fillings and hip prosthesis, would adversely cause the missing of projection data during CT imaging, and thus lead to the obvious streaking artifacts and shadings in the reconstructed CT images. A robust model, which automatically reduces the unsatisfactory metal artifacts and improves the quality of CT images for subsequent clinical treatment, is worthwhile to develop [Wang et al., 2021b; Wang et al., 2021a; Lin et al., 2019].

† Corresponding author

Against this metal artifact reduction (MAR) task, traditional model-based methods focused on reconstructing artifact-reduced CT images by filling the metal-affected region in sinogram with different estimation strategies, such as linear interpolation (LI) [Kalender et al., 1987] and normalized MAR [Meyer et al., 2010]. However, these methods always bring secondary artifacts in the restored CT images, since the estimated sinogram cannot finely meet the CT physical imaging constraint. There is another type of conventional MAR methods, which iteratively reconstruct clean CT images from unaffected sinogram [Zhang et al., 2011] or weighted/corrected sinogram [Lemmens et al., 2008]. Although such model-based methods are usually algorithmically interpretable, the pre-designed regularizers cannot flexibly represent the complicated artifacts in different metalcorrupted CT images collected from real applications.

With the rapid development of deep learning (DL), recent years have witnessed the promising progress of DL for the MAR task. Some early works adopted convolutional neural network (CNN) to reconstruct clean sinograms and then utilized the filtered-back-projection transformation to reconstruct the artifact-reduced CT images [Zhang and Yu, 2018; Ghani and Karl, 2019]. Later, researchers adopted different learning strategies, e.g., residual learning [Huang et al., 2018] and adversarial learning [Wang et al., 2018], to directly learn the artifact-reduced CT images. Very recently, there is a new research line for the MAR task, which focuses on the mutual learning of sinograms and CT images [Lin et al., 2019; Lyu et al., 2020; Wang et al., 2021b; Wang et al., 2021c].

Attributed to the powerful learning ability of CNN, deep MAR methods generally achieve superior performance over conventional model-based approaches. Yet, there is still some room for further performance improvement. First, most of current deep MAR works pay less attention to exploring the intrinsic prior knowledge of the specific MAR task, such as non-local streaking structures of metal artifacts (see Fig. 1). The explicit embedding of such prior information can assist to regularize the solution space for metal artifact extraction, and further boost the MAR performance of deep networks as well as its generalization ability. Second, the involved image enhancement modules in current deep MAR works [Lin et al., 2019; Lyu et al., 2020] are roughly the variants of UNet. With such a general network design, the physical interpretability of every network module is unclear.

In this paper, we propose an explicit model to encode the prior observations underlying the MAR task and fully embed it into an adaptive convolutional dictionary network (namely ACDNet). The proposed framework inherits the clear interpretability of model-based methods and maintains the powerful representation ability of learning-based methods. In summary, our main contributions can be concluded as:

1. Prior Formulation. For the MAR task, we explore that artifacts in different metal-corrupted CT images always present common (i.e., sample-invariant) patterns, such as non-local repetitive streaking structures, but with the specific (i.e., sample-invariant) intensity variation in CT images. Based on such prior observations, we adaptively encode the artifacts in every metal-corrupted CT image as a weighted convolutional dictionary (WCD) model (see Fig. 1).
2. Prior Embedding and Interpretability. To solve the WCD model, we propose an iterative algorithm with only simple operators and easily construct the ACDNet by unfolding every iterative step into the corresponding network module. Similar to learning-based methods, ACDNet automatically learns the priors for artifact-free CT images in a purely data-driven manner, overcoming the disadvantages of handcrafted priors. Similar to model-based methods, ACDNet is explicitly integrated with the WCD model of artifacts and has clear physical interpretability corresponding to our optimization algorithm. The proposed ACDNet thus integrates the advantages of model-based and learning-based methodologies.
3. Fine Generalization. With the regularization of the explicit WCD model, ACDNet can more accurately extract artifacts complying with prior structures. Comprehensive experiments, including cross-body-site generalization and synthesis-to-clinical generalization, finely substantiate the superiority of our method as well as excellent computation efficiency. Especially, as an image-domain-based method without utilizing sinogram, our ACDNet can be adopted as a plugin module, which is easily integrated into current computeraided diagnosis systems to deal with some practical scenarios where the projection data is difficult to acquire.

2 Methodology

In this section, we propose an explicit model to adaptively encode the priors of every metal-affected CT image and derive a simple algorithm for the subsequent network design.

2.1 Weighted Convolutional DictionaryModel

For an observed metal-corrupted CT image Y , it is composed of two regions, i.e., metal part and non-metal part. Since metals generally have higher CT values than normal tissues, we ignore the information in the metal region and make efforts to reconstruct the non-metal region of Y . Therefore, the decomposition model can be derived as:

where I is a binary non-metal mask; X ∈ RH×W is the tobe-estimated clean CT image; A is the to-be-extracted metal artifacts; ⊙ is an element-wise multiplication.

From Eq. (1), the estimation of X and A from Y is an ill-posed inverse problem. Against this issue, most of current learning-based methods empirically design complicated network architectures to directly learn X from Y , which lack the consideration of explicit prior knowledge of MAR (e.g., the patterns of metal artifacts). However, embedding the explicit prior is helpful to finely regularize the solution space of such an ill-posed problem, and thus further improves the performance of deep networks, especially generalization ability. Based on this motivation, we explore the specific prior structures underlying the MAR task, and then propose a strategy to explicitly embed them into deep networks.

Prior Formulation. Specifically, we disclose that for different metal-affected CT images, the metal artifacts are with roughly common patterns, e.g., non-local repetitive streaking structures. Besides, due to the mutual influence between normal tissues and metal artifacts, the patterns of metal artifacts in different CT images are not exactly the same and generally have some specific characteristics, e.g., pixel intensities of metal artifacts vary between different metal-corrupted CT images. Based on these prior observations, we formulate a weighted convolutional dictionary (WCD) model to encode the metal artifacts A as [Wang et al., 2021d]:

where D ∈ Rp×p×d is a sample-invariant dictionary (i.e., a set of convolutional filters) representing common local patterns of metal artifacts in different metal-corrupted CT images; Kn ∈ Rd is a sample-wise weighting coefficient to learn the specific convolutional filter for A by computing D∗Kn as ∑di=1 D[:, :, i]⊙Kn[i] (see Fig. 1); Mn ∈ RH×W is the coefficients representing the locations for the local patterns of metal artifacts; p is the size of convolutional filters; d is the total number of filters in the dictionary D; N is the actual number of filters encoding the artifacts A; and the ⊗ between (D ∗K) and M is a convolutional operation in tensor form. Specifically, (D ∗Kn) ∈ Rp×p , K ∈ Rd×N , (D ∗ K) ∈ Rp×p×N , M ∈ RH×W×N , and

Note that convolutional dictionary model has been verified to be applicable to represent repetitive patterns by existing studies [Wang et al., 2021a]. Compared to these methods, the proposed weighted mechanism in Eq. (2) has two main merits for the MAR task: 1) It delivers not only the sampleinvariant/common knowledge, but also the sample-wise characteristics for information embedding, which can adaptively encode the prior structure for every CT image; 2) With such a weighted convolutional dictionary, we can choose an N smaller than d,* which further shrinks the solution space for estimating A and thereby improves the model generalization (see Sec. 4).

By substituting Eq. (2) into Eq. (1), we can derive:

As shown, our goal is to estimate the sample-wise K, M, and X from Y . Note that the non-metal mask I is pre-known (see Sec. 4) and the common dictionary D can be automatically learnt from training data (see Sec. 3). With the maximum-aposterior framework, the optimization problem is:

where f1 (·), f2 (·), and f3 (·) are regularization functions, delivering the prior knowledge of K, M, and X, respectively; α, β, and γ are regularization weights.

2.2 Optimization Algorithm

Traditional solvers for the problem (5) often involve complex Fourier and inverse Fourier transforms, which are difficult to integrate into deep networks. In this regard, we design a new optimization algorithm with only simple operators. Concretely, the proximal gradient technique [Beck and Teboulle, 2009] is adopted to alternately update K, M, and X:

SolvingK. From the problem (5), at the (t+1)-th iteration, K is solved as follows:

and the quadratic approximation [Beck and Teboulle, 2009] of the objective function in the problem (6) is:

where Ω = {K| ∥Kn∥2 = 1, n = 1, 2, · · · ,N}; g1 (K(t) ) = ∥∥I ⊙ (Y -X(t) - (D ∗K(t) )⊗M(t) )∥∥2F ; η1 is the stepsize. Hence, we can get the following equation:

* In all our experiments below, N = 6 and d = 32.

where ⊗d is the depth-wise convolutional computation; U f 3 (·) denotes the unfolding operation at the 3rd mode; and vec(·) denotes the vectorization.

Clearly, Eq. (7) can be equivalently written as:

For general prior f1(·) [Donoho, 1995], Eq. (9) is derived as:

imal operator which is related to f1 (·). The constraint space Ω can be achieved by introducing a normalization operation into proxαη1 (·) (see Sec. 3).

SolvingM. Similarly, at the (t+1)-th iteration,Mcan be updated by solving the quadratic approximation of the subproblemwith respect toMas:

where η2 is the stepsize and g2 (M(t) ) = ∥∥I ⊙ (Y -X(t) -(D ∗ K(t+1) )⊗M(t) )∥∥2F . Similarly, the solution of problem (11) is deduced as:

where proxβη2 (·) is a proximal operator related to the regularization function f2(·) about M; M(t+0.5) = M(t) -η2∇g2 (M(t) ); ∇g2 (M(t) ) = (D ∗K(t+1) ) ⊗T (I ⊙ ((D ∗K(t+1) ) ⊗M(t) +X(t) -Y )); and ⊗T is the transposed convolution operation.

Solving X. Given K(t+1) and M(t+1) , the quadratic approximation of the subproblem about X is derived as:

(13) where g3 (X(t)) = ∥∥I⊙(Y-X(t)-(D∗K(t+1) )⊗M(t+1) )∥∥2F . Then, the updating rule of X is written as:

where X(t+0.5) = (1-η3I) ⊙ X(t) + η3I ⊙ (Y-(D ∗K(t+1) )⊗M(t+1)); proxγη3 (·) is dependent on the prior function f3(·) about X.

Eqs. (10), (12), and (14) constitute the entire iterative process for solving the problem (5). As seen, the proposed algorithm only contains simple operators, which makes it easier to accordingly build the deep network framework upon the algorithm. Note that proxαη1 (·), proxβη2 (·), and proxγη3 (·) are implicit operators, which are automatically learnt from training data by virtue of the powerful prior fitting capability of deep networks. This manner has been fully used and widely validated to be effective and helpful to explore interpretable knowledge by recent studies (e.g.. [Wang et al., 2021d; Xie et al., 2019]). The details are in Sec. 3.

Figure 2: (a) The proposed ACDNet consists of T stages. (b) The detailed structure at any stage where K(t+1) , M(t+1) , and X(t+1) are successively updated by K-net, M-net, and X-net, respectively, based on Eqs. (10), (12), and (14).

3 Network Design and Analysis

Due to the specific design of joint model-driven (i.e., prior knowledge) and data-driven (i.e., deep networks) frameworks, deep unfolding techniques have achieved great success in computer vision tasks, such as single image rain removal [Wang et al., 2021d] and low-light image enhancement [Liu et al., 2022]. Inspired by this, we specifically design an adaptive convolutional dictionary network (ACDNet) for the MAR task by unfolding the iterative algorithm in Sec. 2.2 into the corresponding network structure.

Fig. 2 (a) displays the entire network structure with T stages, which correspond to T iterations of the optimization algorithm in Sec. 2.2. For every stage, our network consists of three sub-nets, i.e., K-net,M-net, andX-net corresponding to addressing the three subproblems, i.e.,, solvingK,M, and X, respectively. Fig. 2 (b) shows the detailed network connections at every stage, which are constructed by sequentially unfolding the iterative rules, i.e., Eq. (10), Eq. (12), and Eq. (14), respectively. With the unfolding operations, every network module corresponds to the specific iterative step of the proposed optimization algorithm. Thus, the entire network framework has a clear physical interpretability. Next we present the information of sub-nets in detail:

K-net. At the (t + 1)-th stage, K(t+0.5) is computed and fed to proxNetθ(t+1) K (·) to execute the operator proxαη1 (·). Then, the updated weighting coefficient is: K(t+1) = proxNetθ(t+1) K (K(t+0.5) ), where proxNetθ(t+1) K (·) is a residual structure, i.e., [Linear+ReLU+Linear+Skip Connection+Normalization at the dimension d].

M-net. Given K(t+1) , M(t) , and X(t) , we can compute M(t+1) = proxNetθ(t+1) M (M(t+0.5) ), where proxNetθ(t+1) M (·) is the unfolding form of proxβη2 (·)- ResNet with three [Conv+BN+ReLU+Conv+BN+Skip Connection] residual blocks.

X-net. Similarly, given K(t+1) , M(t+1) , and X(t) , the artifact-reduced CT image can be updated by X(t+1) = proxNetθ(t+1) X (X(t+0.5) ), where proxNetθ(t+1) X (·) has the same network structure to proxNetθ(t+1) M (·).

With T-stage optimization, we can get the final CT image X(T) . All the involved parameters are {θ(t) K , θ(t) M, θ(t) X }Tt=1 , {ηi}3i=1 , and the common dictionary D. They can be automatically learnt from training samples in an end-to-end manner. Note that D is achieved by a common convolutional layer. More details, including the initialization (K(0) , M(0) , X(0) ), are given in Supplemental Material (SM).

ProxNet. Following recent great deep unfolding-based works [Wang et al., 2021b; Xie et al., 2019], we also set proximal operator with ResNet. Although it is very hard to inversely derive the form of regularizer by integrating ResNet function due to its complicated form, but as has comprehensively substantiated by previous research, it does be helpful to explore insightful structural prior. Besides, adopting such learning-based manner to automatically learn implicit regularizers is more flexible, avoiding hand-crafted prior design.

Interpretability. Unlike most of current deep MAR networks, which are heuristically built based on the off-the-shelf network blocks, ACDNet is naturally constructed under the guidance of the optimization algorithm with careful data fidelity term design. In this regard, the entire network integrates the interpretability of model-based methods. Besides, such interpretability is visually validated by Fig. 3 of SM.

Remark. Deep unfolding technique is a general tool for network design. To apply it, the key challenge is how to elaborately design model and optimization algorithm to make it work for specific applications. For the MAR task, we have specifically made some substantial ameliorations: 1) The adaptive prior of artifacts is incorporated into data fidelity term which can finely guide the network learning and boost the generalization ability; 2) Unlike other solvers with complicated computations (e.g., Fourier transformation and matrix inversion), the proposed optimization algorithm is elaborately designed which only contains simple operators and makes the unfolding process easy and proper; 3) Compared to current popular dual-domain MAR methods, ACDNet only uses CT image domain data, which is more friendly to practical applications where the projection data is difficult to acquire. Besides, ACDNet can be easily integrated into current computer-aided diagnosis systems as a plug-in module.

Loss Function. With supervision on the extracted artifact A(t) and CT image X(t) at every stage, the training loss is:

where X is the clean (i.e., ground truth) CT image. In all experiments, µT is set to 1; µt (t = 0, 1, · · · , T - 1) is 0.1; ω1 and ω2 are empirically set to 5 × 10-4 .

Implementation Details. ACDNet is optimized through Adam optimizer based on PyTorch. The framework is trained on an NVIDIA Tesla V100-SMX2 GPU with a batch size of 32. The initial learning rate is 2 × 10-4 and divided by 2 at epochs [50, 100, 150, 200]. The total number of epochs is 300. The size of input image patch is 64 × 64 pixels and it is randomly flipped horizontally and vertically. More explanations are included in SM.

4 Experiments

In this section, we conduct extensive experiments to validate the effectiveness of our method. †

† More results, including interpretability verification, ablation study, convergence process, and P-values analysis are given in SM.

4.1 Dataset &Experimental Setting

Synthesized Data. Following the simulation procedure in [Yu et al., 2020], we randomly choose 1,200 clean CT images from the public DeepLesion dataset [Yan et al., 2018] and collect 100 metal masks from [Zhang and Yu, 2018] to synthesize the paired clean/metal-corrupted CT images. Specifically, 90 metals together with 1000 clean CT images for training and 10 ones together with the remaining 200 clean CT images for testing. Similar to [Lin et al., 2019; Lyu et al., 2020], we sequentially take every two testing metal masks as one group for performance evaluation. In addition, to evaluate the cross-body-site generalization performance, clean dental CT images [Yu et al., 2020] are adopted and the corresponding metal-corrupted images are generated under the same simulation protocol on DeepLesion.

Clinical Data. A public clinical dataset, i.e., CLINICmetal [Liu et al., 2021], which contains 14 metal-corrupted volumes with pixel-wise annotations of multiple bone structures (i.e., sacrum, left hip, right hip, and lumbar spine), is used for evaluation. Following [Yu et al., 2020], the clinical metal masks are segmented with thresholding of 2500 HU.

Baselines. Representative MAR methods are used, including traditional LI [Kalender et al., 1987] and NMAR [Meyer et al., 2010], learning-based CNNMAR [Zhang and Yu, 2018], DuDoNet [Lin et al., 2019], DSCMAR [Yu et al., 2020], and DuDoNet++ [Lyu et al., 2020].

Evaluation Metric. We adopt the PSNR/SSIM for quantitative comparison on synthesized data and only visual comparison on clinical data due to the lack of clean CT images.

(a)

(b)

Ground Truth Input NMAR

CNNMAR

DuDoNet DSCMAR

DuDoNet++ ACDNet (Ours)

Figure 4: Generalization results. Artifact-reduced results on the synthesized dental CT images with different metallic implants.

4.2 Performance Evaluation

DeepLesion Data. Fig. 3 presents the visual comparison on an image from synthesized metal-corrupted DeepLesion. As shown, our approach significantly removes the artifacts and finely recovers evident details. The average PSNR/SSIM scores on the entire DeepLesion dataset are listed in Table 1. It can be observed that ACDNet consistently achieves the best scores with varying sizes of metallic implants.

Dental Data. Fig. 4 shows the MAR visual results on the synthesized dental CT images with different metallic implants, where deep MAR methods are trained on the synthesized DeepLesion (focusing on abdomen and thorax) and directly tested on the dental CT images. Due to the regularization with the explicit WCD model, ACDNet can more accurately identify the artifacts and accomplish the better reconstruction of artifact-reduced CT images. The results show the excellent cross-body-site generalizability of our method.

Clinical Data. Fig. 5 presents theMAR comparison of generalization results on a clinical metal-corrupted CT image. Our ACDNet finely preserves more bone structures and removes more artifacts. Fig. 6 displays the downstream segmentation results of an artifact-reduced CLINIC-metal volume, which are reconstructed by different MAR approaches. The proposed ACDNet is significantly superior to other approaches, which reveals its excellent potential for clinical applications. Due to the limited space, we only provide the results of the latest MAR methods.

Computation Efficiency. Table 2 reports that our method has fewer network parameters (i.e., storage cost) and computation consumption, compared to other methods.

Table 2: The number of network parameters and average testing time (seconds) computed on 2000 images with size 416 × 416 pixels on an NVIDIA Tesla V100-SMX2 GPU.

5 Conclusion and FutureWork

In this paper, for the MAR task, we have proposed a weighted convolutional dictionary model to explicitly and adaptively encode the structural prior of metal artifacts for every input image. Then, by unfolding a simple-yet-effective optimization algorithm, we have easily built the deep network, which integrated the advantages of both model-based methods and learning-based methods. Experiments executed on three public datasets verified the excellent generalizability and excellent computational efficiency of our method.

Following current state-of-the-art (SOTA) MAR methods, we adopt the thresholding manner to simply segment the metal for clinical data. An unsatisfactory threshold possibly makes tissues be wrongly regarded as metals and most MAR methods would fail to recover image details. Although our method consistently achieve good performances for different sizes of metals and show the robustness, as a direction to further boost the performance of our framework, the joint optimization of automated metal localization and MAR is worthwhile to further explore. Besides, like SOTAMAR methods, we process CT images slice by slice for fairness. It would be very meaningful to do 3D prior modeling for the MAR task.

Acknowledgments

This work was founded by the China NSFC project under contract U21A6005, the Macao Science and Technology Development Fund under Grant 061/2020/A2, the major key project of PCL under contract PCL2021A12, the Key-Area Research and Development Program of Guangdong Province, China (No. 2018B010111001), National Key R&D Program of China (2018YFC2000702), the Scientific and Technical Innovation 2030-"New Generation Artificial Intelligence" Project (No. 2020AAA0104100).