Degradation Accordant Plug-and-Play for Low-Rank Tensor Completion

1 Introduction

The main aim of tensor completion is to fill the missing values of the data in the tensor format when the observation is incomplete owing to objective conditions. It has been widely studied for many real-world applications, such as, color image and video processing [Liu et al., 2013], personalized web search [Sun et al., 2005], high-order web link analysis [Kolda et al., 2005], and fine-grained indoor localization [Liu et al., 2015]. As many types of real-world highdimensional tensor data, e.g., imaging data, seismic data, and traffic data, maintain low-dimensional structures, the tensor low-rankness is widely utilized for the tensor completion [Liu et al., 2013], and low-rank tensor completion (LRTC) methods have achieved great successes in the past decade.

∗Corresponding author.

In this work, we mainly focus on the completion of multidimensional imaging data, including color images, videos, magnetic resonance imaging (MRI) data, multispectral images (MSIs). Meanwhile, although the rank of the tensor is not uniquely defined, we fix our attention on the tensor tubal rank defined based on the tensor singular value decomposition (t-SVD) [Kilmer and Martin, 2011] framework since that the t-SVD is specialized for third-order (namely, cubic) tensors, being suitable for multidimensional imaging data, and the data would be processed integrally within the t-SVD algebraic framework avoiding the loss of information inherent in matricization [Kilmer et al., 2013]. Zhang et al. minimize the tensor tubal nuclear norm (TNN) [Zhang and Aeron, 2017], which is a convex surrogate of the tensor tubal rank, for LRTC and given the theoretical guarantee for exact recovery.

As we know, multidimensional images reserve abundant spatial details, making themselves not strictly low-rank. Thus, to better address this inverse problem, additional prior knowledge is needed to better preserve those details. For example, the total variation (TV) regularizer is introduced to characterize the piece-wise local smoothness in [Jiang et al., 2018]. More recently, Zhao et al. [Zhao et al., 2020] formulate an implicit regularizer and adopt a deep convolution neural network (CNN), which is pretrained on a large number of natural images for denoising, to express the deep denoising prior under the plug-and-play (PnP) framework [Venkatakrishnan et al., 2013].

Within the flexible PnP framework, the degradation and the prior knowledge can be well decoupled after variable splitting via half-quadratic splitting [Nikolova and Ng, 2005] or the alternating direction method of multipliers (ADMM) [Boyd et al., 2011], and one can directly employ off-the-shelf denoising algorithms to express the prior knowledge for various image inverse problems, such as deblurring and superresolution. When the well-known BM3D [Dabov et al., 2007] is plugged in, it is believed to enhance the nonlocal self-similarity. With the rapid development of deep learning, we can also use deep CNN denoisers to utilize the data-driven prior learned from a mass of training images. Thanks to the high model capacity of the CNN, the method in [Zhao et al., 2020] achieves promising performance for tensor completion.

We would like to take a further step to plug in a CNN inpainter for LRTC in this work. Two facts motivate us to conduct this.First, the plugged-in inpainter is designed for the inpainting task, which is naturally consistent with the degradation in tensor completion, being more suitable than denoisers. Second, as the model capacity of the CNN is quite high, it is reasonable to resort to the plugged-in CNN for undertaking parts of the degradation as well as expressing the image prior. Thus, our model is given by combining the internal low-rank prior with the external degradation accordant datadriven prior together. On the one hand, the TNN regularizer characterizes the global low-dimensional structure and inherent connection of the multi-dimensional data. On the other hand, an off-the-shelf CNN, which is pretrained for the image inpainting task on images, is employed to preserve the spatial details. Our contributions are summarised as follows.

• A novel degradation accordant PnP LRTC model is proposed for multidimensional images. In our model, the TNN is utilized to depict the inherent global low-rank structure of underlying multidimensional images. An implicit regularizer is formulated to better preserve the abundant details in multidimensional images via plugging in a CNN image inpainters. Those two terms are organically complementary to each other and our model is therefore expected to be effective.
• We customize the ADMM for the proposed model and the subproblem corresponding to the implicit regularizer is solved via the CNN inpainter, which is in accord with the original degradation process. This inpainter is pretrained on natural images, which are readily accessible, to faithfully express the data-driven image prior. Numerical experiments are conducted on various types of multidimensional images. Comparisons with state-of-the-art methods illustrate the excellent performance and generalization ability to different types of data of our method.

The outline of this paper is given as follows. Sec. 2 gives the basic preliminaries of the tensor. Sec. 3 presents main results. Sec. 4 illustrates experimental results. Finally, conclusions are drawn in Sec. 5.

2 Preliminaries

Throughout this paper, lowercase letters, e.g., x, boldface lowercase letters, e.g., x, boldface uppercase letters, e.g., X, and boldface calligraphic letters, e.g., X, are used to denote scalars, vectors, matrices, and tensors, respectively. Given a third-order tensor X ∈ Rn1×n2×n3 , we use Xijk to denote its (i, j, k)-th element. The k-th frontal slice of X is denoted as X (k) (or X(:, :, k), Xk ). We use the notation ̂ A to denote the Fourier transformed (along the third mode) tensor of A. The conjugate transpose of a tensor A ∈ Cn2×n1×n3 is tensor AH ∈ Cn1×n2×n3 obtained by conjugate transposing each of the frontal slice and then reversing the order of transposed frontal slices 2 through n3.

Definition 1 (T-prod [Kilmer andMartin, 2011]). The tensortensor-product (t-prod) C = A ∗ B of A ∈ Rn1×n2×n3 and B ∈ Rn2×n4×n3 is a tensor of the size n1×n4×n3, where the (i, j)-th tube cij: is given by C(i, j, :) = ∑n2 k=1 A(i, k, :) ⊛ B(k, j, :), where ⊛ denotes the circular convolution between two tubes of the same size.

Definition 2 (Special tensors [Kilmer and Martin, 2011]). The identity tensor I ∈ Rn1×n1×n3 is the tensor whose first frontal slice is the n1 × n1 identity matrix, and whose other frontal slices are all zeros. A tensor Q ∈ Cn1×n1×n3 is orthogonal if it satisfies QH ∗ Q = Q ∗ QH = I. A tensor A is called f-diagonal if each frontal slice A(i) is a diagonal matrix.

Theorem 1 (T-SVD [Kilmer and Martin, 2011]). For A ∈ Rn1×n2×n3 , the t-SVD of A is given by

where U ∈ Rn1×n1×n3 and V ∈ Rn2×n2×n3 are orthogonal tensors, and S ∈ Rn1×n2×n3 is an f-diagonal tensor.

Definition 3 (Tensor tubal-rank [Zhang and Aeron, 2017]). The tubal-rank of a tensor A ∈ Rn1×n2×n3 , denoted as rankt(A), is defined to be the number of non-zero singular tubes of S, where S comes from the t-SVD of A: A = U ∗ S ∗ V ⊤.

Definition 4 (Tubal-nuclear-norm (TNN) [Zhang and Aeron, 2017]). The tensor nuclear norm of a tensor A ∈ Rn1×n2×n3 , denoted as ‖A‖TNN, is defined as

where ‖ · ‖∗ is the matrix nuclear norm.

3 Main Results

Our LRTC model is formulated as

where A is a linear degradation operator, O ∈ Rn1×n2×n3 is the observed multidimensional image 1 and Φ(·) is an implicit regularizer. We introduce two auxiliary variables and reformulate (3) as

Generally, one auxiliary variable is enough to decouple the low-rank regularizer and the implicit regularizer. We adopt two to formulate the implicit regularizer related subproblem in the image restoration format instead of denosing. This makes our method different from previous PnP methods.

Then, the augmented Lagrangian function is

where Λis (i = 1, 2, 3) are the Lagrangian multipliers, and βis (i = 1, 2, 3) are nonnegative parameters. Next, we set up ADMM iterations via solving following subproblems.

1 The spatial resolution is n1 ×n2 and the spectral (or temporal) length of the multidimensional image is n3.

Z-subproblem: At the k-th iteration, the subproblemwith respect to Z is

Let the t-SVD of (Y k + Λk3 β3 ) be U ∗ S ∗ V H , the closed-form solution of (5) is

where D is an f-diagonal tensor obtained satisfying ̂D(i, i, k) = max{ ̂ S(i, i, k) - λ n3β3 , 0}.

Y-subproblem: At the k-th iteration, the subproblemwith respect to Y is

Denoting the adjoint operator of A as A∗ , we can obtain the solution of (7) as

where I denotes the identity mapping.

X-subproblem: At the k-th iteration, the subproblemwith respect to X ismin

where B = (β1O-Λk1 +β2A(Y k+1 )-Λk2) β1+β2 and Φ′ (X) = Φ(X) β1+β2 . We further decouple (8) into n3 subproblems as

for i = 1, 2, · · · , n3. Thus, it becomes n3 image restoration problems and can be solved via off-the-shelf algorithms under the PnP framework. For the tensor completion, the degradation operator is indeed the projection operator PΩ(·), which keeps the entries in Ω and set remaining entries as 0. We can solve it using inpainting algorithms as

for i = 1, 2, · · · , n3.

Finally, the update of multipliers follows the standard ADMM and can be seen in Supplementary Material 2 .

Remark: Our method is not limited to the tensor completion. Other multidimensional image restoration tasks, such as video superresolution or deblurring, can also be addressed via replacing the degradation operator A(·) and adopting corresponding image restoration algorithms in (9). We would like to utilize CNNs in (9). As analyzed in [Ryu et al., 2019], the convergence of our algorithm would be guaranteed if the CNN is properly trained. Meanwhile, we want to also emphasize the generalization ability of our method as our method can be applied for different types of multidimensional images and only needs the inpainting CNN trained on grayscale or color images.

2 Can be found at https://github.com/TaiXiangJiang/.

4 Experiments

In this section, we evaluate the effectiveness of our degradation accordant PnP LRTC (DAP-LRTC) method and compare it with other state-of-the-art methods on color images, videos, MSIs, and MRI data. Compared methods consist of: the Tucker-rank based method HaLRTC 3 [Liu et al., 2013], a t-SVD based method (TNN) 4 [Zhang and Aeron, 2017], a DCT induced TNN minimization method (DCTNN) 5 [Lu et al., 2019], a framelet represented TNN minimization method (FTNN) 6 [Jiang et al., 2020], a deep denoiser regularized TNN minimization method (DP3LRTC) 7 [Zhao et al., 2020], and a deep video inpainter called Onion-peel networks (OPN) 8 [Oh et al., 2019].

For all experiments, two numerical metrics are employed, including the Peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM) [Wang et al., 2004]. Higher PSNR and SSIM values mean better performance. Additionally, we introduce the mean spectral angle mapper (SAM) for MSIs, and lower SAM indicates better results. All experiments were conducted on the platform of Window 10 with an AMD Ryzen9 3950X CPU and RTX 2080Ti GPU and 32RAM. 2 CNN inpainters are considered in our method: i) Deepfillv2 9 [Yu et al., 2019] for structural missing, ii) CRUNet (we borrow the network structure of DRUNet 10 in [Zhang et al., 2021] and train it for grey-scale image completion) for randommissing. The training details of CRUNet are given in Supplementary Materials. The results by applying Deepfillv2 and CRUNet on each band of the incomplete data would also be reported for reference.

4.1 Color Image

In this subsection, 6 color images 11 of the size 512 × 512 × 3 are selected. We consider the structural missing in all RGB channels. We adopt deepfillv2 to solve (9). Tab.1 reports quantitative metrics and the running time in seconds. We can see that our method outperforms compared methods. Fig.1 shows the inpainting results by different methods on color images (Baboon, Painting, Lena) with three different masks, respectively. From the enlarged areas, the superior of our method is obvious. More results on color images are shown in Supplementary Material.

3 https://www.cs.rochester.edu/∼jliu/code/TensorCompletion.zip

4 https://github.com/jamiezeminzhang/Tensor Completion and Tensor RPCA

5 Implemented by ourselves based on the code of TNN

6 https://github.com/TaiXiangJiang/Framelet-TNN

7 https://taixiangjiang.github.io/

8 https://github.com/seoungwugoh/opn-demo

9 https://github.com/JiahuiYu/generative inpainting

10 https://github.com/cszn/DPIR

11 Available at http://sipi.usc.edu/database/database.php.

4.2 Video

In this subsection, we choose four videos 12 to test the capacity of the proposed method. The size of all videos is 144 × 176 × 50. For videos, we consider both the random missing and structural missing. Deepfillv2 and CRUNet are respectively employed for these two cases. Tab.2 exhibits the quantitative metrics and the running time on the videos Bridge-close and Bridge-far for structural missing. It can be observed that our method gets the highest metrics. The remainder of the experiments on video data is also given in the supplementary material. The visual results on the video Bridge-close are shown in Fig.2. We can see that our method and Deepfillv2 preserve the structure of the bridge while other methods failed. We further plot one temporal vector of the missing area in Fig.3. From Fig.3, it can be seen that the temporal vector of our result is the closest to the original data. This shows the effectiveness brought by the TNN regularizer and validates that the TNN and the CNN inpainter are complementary to each other. Please see Supplementary Material for more results on videos.

12 Videos available at http://trace.eas.asu.edu/yuv/.

4.3 MSI

In this part, we select 4 MSIs 13 of the size 512 × 512 × 31. The observation is obtained via random sampling with the sampling rates (SRs) 0.03 and 0.05, respectively. We use CRUNet in our method for this random missing case. In Tab.3, we list quantitative metrics of results by different methods on MSI Thread spool with different SRs. We can see that the performance of FTNN, DP3LRTC, and CRUNet are good as they occasionally achieve the second best. Although the inpainter is not trained for MSIs, our method obtains the best values for different metrics and sampling rates, showing remarkable generalization ability. The pseudo-color images (composed of three bands of the MSI) of the reconstructed results by different methods are displayed in Fig.4. We can see that OPN, which is trained for videos could not handle the MSI. Our method, DP3LRTC, and CRUnet obtain good results while we can see over-smoothness in the result by DP3LRTC and the color distortion in the result by CRUnet. More experimental results are in Supplementary Material.

13 http://www.cs.columbia.edu/CAVE/databases/multispectral/.

4.4 Parameter Analysis and Convergence

There are four parameters, i.e., λ, β1, β2, and β3, in our method. To test the effects from different values of them, we conduct experiments on the MRI 14 data with the random sampling rate of 10%. When testing one parameter, the other three are fixed as default values. We illustrate the PSNR and SSIM values with respect to different values of those parameters in Fig. 5. From Fig. 5, we can see that the performance of our method is more sensitive to λ and β3.

14 Available at https://brainweb.bic.mni.mcgill.ca/brainweb. Please refer to Supplementary Material for more results on MRI data.

Also on the MRI data with random sampling rate 10%, we report the relative change of each variable with respect to iteration numbers in Fig. 6. Meanwhile, the result on the color image "baboon" with structural missing (first row of Fig. 1) is also plotted. We plug in the CRUnet for the MRI data with random missing and Deepfillv2 for the color image with structural missing. We can see the empirical convergence of our method for different settings.

5 Conclusion

We propose a novel degradation accordant PnP LRTC approach for multidimensional images. In our model, the TNN is utilized to depict the global low-rank structure of underlying multidimensional images. An implicit regularizer is formulated and its corresponding subproblem is equivalent to a series of single image restoration problems. The CNN pretrained for image inpainting is plugged in as the solution. It is consistent with the original degradation process and faithfully expresses the image prior learned from a large number of training images. An ADMM algorithm is tailored for our model. Numerical experiments are conducted on various types of multidimensional images to compare our method with state-of-the-art methods, illustrating the excellent performance of our method. Although our method obtains the best performance and shows remarkable generalization ability for different types of data, it is still time-consuming. This would be considered in future work.

Acknowledgements

We would like thank the authors of [Liu et al., 2013; Zhang and Aeron, 2017; Lu et al., 2019; Zhang et al., 2021] for their generous sharing of their codes and data. This research is supported by National Natural Science Foundation of China (Nos. 12001446, 61876203, 61772003), the Key Project of Applied Basic Research in Sichuan Province (No. 2020YJ0216), the Applied Basic Research Project of Sichuan Province (No. 2021YJ0107), National Key Research and Development Program of China (No. 2020YFA0714001), the Fundamental Research Funds for the Central Universities under Grant (Nos. JBK2202049, JBK2102001).