QFormer: An Efficient Quaternion Transformer for Image Denoising

1 Introduction

Image denoising aims to remove noise from the noisy images to recover high-quality images. At the same time, the image denoising task is an ill-posed problem, thus it is extremely challenging. Although the image denoising performance of Deep Convolutional Neural Networks (DCNNs)-based methods have been significantly improved compared to traditional model-based methods [Joshi et al., 2005], they usually suffer from a critical problem in that vanilla convolution retrieves local spatial information but severely lacks the long-range dependencies. Transformer [Kolesnikov et al., 2021], an alternative to DCNNs, can capture long-range pixel dependencies. This drives the development of Transformer-based methods [Liang et al., 2021] for image denoising, achieving state-ofthe-art performances.

∗Corresponding author

Although both DCNNs and Transformer can bring powerful generalization capability to image denoising models, they severely suffer from two main problems. (a) The convolution operator and Transformer only simply add the outputs along the channel axis, resulting in ignoring the complicated interrelationship among channels. The vanilla convolutional layer in DCNNs and the input/output projection layer in Transformer both independently process each color channel, which is also known as a monochromatic model. A significant limitation of the monochrome model is that the correlation among the three color channels is completely retrieved. This probably leads to color distortion within denoised color images. Concatenation models are proposed to simply weigh the three color channels to exploit channel correlation [Xu et al., 2017]. This approach may not adequately explore the complex inter-relationships among channels. Hence, existing denoising methods are not effective to model such important color structural information i.e., the correlation information among the three color channels. (b) Additionally, DCNNsbased or Transformer-based image denoising models are usually inefficient with considerable parameters, high computational complexity, and slow inference speed. Although DCNNs can significantly improve image denoising performances with increasing depth and width (number of channels), it can cause a corresponding increase in parameters and computational cost. Furthermore, the main computational overhead of Transformer-based methods comes from the selfattention mechanism, and its complexity grows quadratically with the spatial resolution, leading to a serious inefficiency in processing high-resolution images. Therefore, it is urgent to study an efficient image denoising method considering the complex mutual information along the channel axis.

Quaternion algebra provides an elegant way of working with vector signals [Moxey et al., 2003]. Since the quaternion unit contains one real part and three imaginary parts, color images can be represented as quaternion matrices, enabling multi-channel information to be processed in a parallel manner. Hence, the quaternion transformation is exactly suitable for color image denoising tasks. Considering the powerful representation of color images by quaternions and the powerful ability of modeling long-range pixel dependencies by Transformer, the key difficulties and challenges in designing an efficient image denoising network based on quaternions and Transformer are mainly from two aspects.

(1) Primitive quaternion operator is non-strict identity mapping for the image denoising task. Strict identity mapping can not only preserve overall texture and color information but also generate high-frequency detail information in image processing methods [Song et al., 2021]. From a mathematical point of view, we have demonstrated that primitive quaternion operators embedded in deep neural networks for image denoising tasks are non-strict identity mappings. Therefore, this becomes a challenge to the application of the primitive quaternion operator in the image denoising tasks. To overcome this limitation, we modify the primitive quaternion operator by using short skip connections in the quaternion operator to alleviate this problem.

(2) How to redesign the Transformer framework with a simple structure, low computational complexity, few parameters, and fast inference speed. We propose the core Quaternion Transformer Block (QTB) integrating the improved quaternion operator with strict identity mapping to modularly formulate our framework of highly-efficient Quaternion Transformer (QFormer) Network to capture both longrange dependencies and local context features with abundant color structure information. The proposed QTB reduces the computational complexity through avoiding the element-wise multiplications in computing self-attention matrices. Furthermore, our QTB is further simplified from a multi-branch structure to an elaborately sequential structure. Due to these mechanisms, the proposed QTB can significantly reduce the computational complexity and especially improve the practical inference speed.

We believe that our theoretical statements and empirical results lay the foundation for new Transformers in the supercomplex domain. These Transformers can grasp the internal input relationship and reduce the computational cost. As far as we know, this is a promising exploration that a Transformer framework has been defined in a super-complex domain. The contributions of this paper can be summarized as follows:

• We propose a highly-efficient Quaternion Transformer (QFormer) Network using Quaternion Transformer Block (QTB) for the image denoising task. The proposed QTB alternately captures both the long-range dependencies and local contextual features by sufficiently retrieving the color structure information.
• We analyze that quaternion operators are not strictly identically mapped in embedding deep neural networks for image denoising tasks from the mathematical perspective, and thus propose utilizing short skip connections to alleviate this obstacle.
• In terms of model efficiency, the proposed sequentialstructured QTB is concise and can avoid considerable element-wise multiplications of computing selfattention matrices, significantly reducing the computation complexity and improving the practical inference speed.
• The proposed QFormer has produced state-of-the-art results from extensive and comprehensive experiments on image denoising tasks, demonstrating satisfactory superiorities in both the denoising performance and efficiency.

2 RelatedWork

2.1 Quaternion-BasedMethods

Since quaternion-based methods can well characterize the cross-channel correlation of color images, they have attracted increasing attention in various vision tasks, such as vector projection [Ell and Sangwine, 2007], and face recognition [Zou et al., 2016], image denoising [Xu et al., 2015]. A model for sparse representation of the quaternion vector was proposed in [Xu et al., 2015] for color image denoising. Several quaternion wavelet methods [Yin et al., 2012] were proposed to achieve a more efficient spatial-spectral image analysis.

The Quaternion Convolutional Neural Network (QCNN) [Zhu et al., 2018] was proposed to extend the real-valued convolutional networks to the complex-valued convolutional networks. Zeng et al. [Zeng et al., 2016] proposed a color image classification network based on quaternary principal component analysis. Parcollet et al. [Parcollet et al., 2019] proposed studying the influence of the Hamilton product on the task of color image reconstruction based on QCNN only from grayscale images. In general, the main advantage of quaternions for color image representations over vanilla convolutional methods is generalization ability and low computational cost [Xu et al., 2015; Parcollet et al., 2019].

2.2 Vision Transformer

Transformer was originally proposed by [Vaswani et al., 2017] for Natural Language Processing (NLP) tasks, and it achieved amazing performances in language pre-training tasks [Devlin et al., 2018]. Inspired by the application of Transformer in NLP tasks, it has been applied to numerous vision tasks such as image recognition [Kolesnikov et al., 2021], segmentation [Wang et al., 2021a], and object detection [Carion et al., 2020]. Vision Transformer (ViT) [Kolesnikov et al., 2021] embedded hard patches (16 × 16) to compute correlation and achieved excellent results in image classification. However, the computational complexity of self-attention in Transformer is positively correlated with the square of the number of image patches, resulting in high computational complexity on high-resolution image reconstruction tasks [Liang et al., 2021]. For example, superresolution [Liang et al., 2021], image colorization [Kumar et al., 2021], denoising [Wang et al., 2021b], and deraining [Wang et al., 2021b]. Much work has focused on improving the transformer's self-attention methods by means of shifting windows [Liang et al., 2021], relative position encoding [Wu et al., 2021b], combining convolutions [Wu et al., 2021a], etc. However, just utilizing pooling as an alternative to the basic self-attention in MetaFormer [Yu et al., 2021] can still achieve competitive or even state-of-the-art performance. This implies that the most effective part in Transformer may be the Transformer-unique framework rather than the selfattention part.

3 Some Preliminaries

In this section, to easily understand the analysis, we mainly define the mathematical notations and preliminaries of quaternion algebra. Following [Chen et al., 2020], scalars, vectors, matrices, and tensors are denoted as lowercase letters, boldface lowercase letters, boldface capital letters, and boldface italic script letters, e.g., x, x, X, and X, respectively.

3.1 Quaternion Representation of Color Images

Quaternion was first proposed by Hamilton in 1843 [Hamilton, 2022] as an extension of real space R and complex space C to describe the position information of points in threedimensional space. The definition [Hamilton, 2022] of a quaternion ˙p ( ˙p ∈ H) has one real part and three imaginary parts as follows:

where q0, q1, q2, q3 (q0, q1, q2, q3 ∈ R) are the coefficients of the real and imaginary parts, respectively. i, j, k are three imaginary units obeying the quaternion rules that i2 = j 2 = k2 = ijk = -1. In addition, if the real part q0 = 0, ˙p is called pure quaternion.

Let ˙Iq be an RGB image matrice with the quaternion representation. RGB channel matrices can be represented as a pure quaternion:

where R,G and B indicate the red, green and blue channel of the color image, respectively.

To directly establish the relationship among the R,G and B channels, as shown in Fig. 1, we use a quaternion rotation matrix to closely connect the three channels of the color image represented by pure quaternions. Using the Rodrigues formula [Murray et al., 1994], the quaternion rotation matrix Q can be obtained from Eqn. 2:

Furthermore, in order to compute the R,G and B channels of the reconstructed color image from the quaternion rotation matrix, we used Eqn. 4 to compute the information for the

three channels:

where Qi,j represents the element at the ith row and the j th column in quaternion rotation matrix Q.

3.2 IdentityMapping of QNNs

The essence of image denoising methods based on traditional end-to-end DCNNs is to learn the mapping relationship from noisy images to clean images. Since the difference between the input and output feature maps of each convolutional layer is very small, this phenomenon is called strict identity mapping, which is greatly important for deep learning-based image processing methods [Song et al., 2021]. Strict identity mapping not only preserves overall texture and color information but also produces high-frequency detail information. Although the mutual information among the color channels can be sufficiently preserved by the quaternion neural networks, it cannot always produce high-frequency detail information, such as the relation described in Theorem 1 below.

Theorem 1. For the arbitrary noisy image ˙In, and an quaternion filter with its weight parameters ˙W. When∆ζ = 0, there exists a ˙Wthat satisfies Eqn. 5, which means that the quaternion neural network cannot produce high-frequency detail information.

where ∆ζ denotes the high-frequency detail information and ⊛ represents the quaternion convolution operation defined in [Zhu et al., 2018]. Eqn. 5 can be rewritten as Eqn. 6 according to the above constraints:

Proof. Here we consider a quaternion fully-connected layer for simplicity. Furthermore, the following proof can be easily extended to quaternion convolution layers. Then, let ˙In be the identity quaternion matrix, we have:

where ˙1 is an identity matrix whose elements are all 1. Then,

where ˆai = 1 denotes an element of the identity quaternion matrix ˙In, si > 0 (si ∈ R) is the magnitude of ˆwi, and N is the number of elements in the quaternion matrix. According to literature [Zhu et al., 2018], ˆwi follows the quaternion convolution weight:

where θi ∈ [-π, π] is the rotation angle parameter of the quaternion convolution operation. ϵ is the unit basis vector, i.e., ϵ = √3 3 (i + j +k).

Then we have:

Since the value range of [1-2 (sin θi 2 )2 ϵ2 ] is [-1, 1], combining Eqns. 7 and 10, we have 1N ∑Ni=1 1 si ˆwi ˆw∗ i = ˙1, which is obviously possible.

According to the above theorem and analysis, although the quaternion neural networks can fully retain the mutual information of the color channels, they cannot always produce high-frequency detail information. This indicates the primitive quaternion operator lacks a strict identity mapping relationship. While stacking more quaternion layers may alleviate this problem, it can significantly increase the model size and computational cost. To address this issue, in practice, we provide a short skip connection operation for quaternion convolutional units or fully connected units, i.e.:

where ˙Wm is the weights of quaternion filters in the mth layer, ˙Imx and ˙Imy are the input and output data, respectively.

Considering ˙Wm ⊛ ˙Imx as ∆ζ, that is, high-frequency detail information, according to the analysis of Theorem 1, the short skip connection operation can effectively alleviate the shortage of strict identity mapping of quaternion operators.

4 QFormer Networks

4.1 Quaternion Transformer Block (QTB)

As shown in Fig.1, the overall structure of the proposed QFormer is a symmetric network mainly composed of encoder and decoder, with the Quaternion Transformer Block (QTB) as its core component. As shown in Fig.2, QTB is an elaborately sequential structure mainly with quaternion transformations. Our QTB and the multi-head self-attention within the traditional Transformer are different in the following two aspects.

(1) QTB captures long-range dependencies also containing local features with abundant color structure information. Capturing long-range dependencies aims to improve the receptive field of deep neural networks to improve model performance. The self-attention mechanism brute forcibly captures long-range dependencies by computing the interaction between any two patches using element-wise multiplication. Diametrically, our QTB first refines global features through several fully connected layers based on the quaternion operator and GELU activation function for the purpose of global information extraction. Then the quaternion operator is used to further extract local features with color structure information from the refined global features to obtain local information. Finally, the produced global information is supplemented with the local information by element-wise aggregating them together using skip connections. Such captured long-term dependencies contain abundant local features with color structure information.

(2) The proposed QTB is more efficient in both computational complexity and practical inference speed than that of the self-attention mechanism. Because the selfattention mechanism adopts element-wise multiplication to capture long-range dependencies, its computational complexity increases quadratically with the size of the input feature. This greatly increases the training and inference costs of the model. Compared with the self-attention mechanism, the proposed QTB uses element-wise addition to aggregate global features with local features to capture long-range dependencies to avoid excessive computational complexity. Therefore, our QTB can significantly reduce the computational complexity. Furthermore, the sequential structure of the proposed QTB can further promote the practical inference speed, evaluated in Section 5.4.

Specifically, the QTB input ˙Fin ∈ HH×W×C is first reshaped to tokens, which are fed into a quaternion fully connected layer to extract global features, as shown in Eqn. 12:

where fqfc denotes quaternion fully connected layer, and ˙Fc ∈ HHW×C is the global features. Then, after normalizing ˙Fc, we use the quaternion linear layer and the GELU nonlinearity activation layer to further capture the global feature. The computation of the above process is represented as:

where LN represents the layer normalization [Ba et al., 2016], f 1 ql and f 2 ql both are quaternion convolutional layers, and ϕ denotes GELU non-linearity activation layer [Hendrycks and Gimpel, 2016].

To enhance the ability of QTB to capture local contextual feature, several quaternion convolutional layers are attached to process ˙Fmid ∈ HH×W×C . As shown in Fig. 2, we first apply a normalization layer to each token. Next, the tokens are reshaped into 2D feature maps, and then three sets of quaternion convolutions with the kernel size of 3 × 3 and GELU activation layers are employed to capture local information. Then, the local features are flattened into tokens and the number of local feature channels is reduced to the same number as the global feature channels using a quaternion linear layer. Finally, element-wise addition of global and local features is conducted to capture long-range dependencies.

4.2 Loss Function

The Charbonnier loss is applied as the ultimate loss function for the proposed QFormer, formulated as follows:

where Nt denotes the number of training samples, Itraget represents the groundtruth corresponding to the input noise image, ϵ2 is a constant that is empirically set to 1×10-6 .

5 Experiments

5.1 Architecture Scales

In order to illustrate the high efficiency of the proposed QFormer in image denoising, three different scales of parameters for QFormer are constructed in our experiments, i.e., QFormer-T (Tiny, C = 16), QFormer-S (Small, C = 32) and QFormer-B (Base, C = 44). The difference between the above settings is only the number of feature channels C, and other settings remain unchanged, such as the depth N of QTB is set to 2.

5.2 Evaluation on Real-world Noisy Images

The QFormer's performance on real-world noisy images, containing complex and unknown noise, is evaluated, highlighting its practical value in real-world denoising applications. Table 1 presents the results obtained from denoising real noisy image datasets. The proposed QFormer is subject to a comparative analysis against fourteen state-of-the-art denoisingmethods. It is evident that the proposed QFormer significantly enhances the PSNR/SSIM results when compared to the other fourteen state-of-the-art methods across the three real noisy image datasets. In the case of the SIDD dataset, QFormer-T demonstrates similar gains in PSNR and SSIM as Uformer and Restormer. Furthermore, it achieves competitive performance alongside HPDNet on the PolyU datasets and with APD-Nets on the Nam datasets. Additionally, QFormerS surpasses HPDNet and Restormer with an average PSNR improvement of 0.26 dB on PolyU and 0.04 dB on SIDD, respectively. Moreover, QFormer-B outperforms APD-Nets by an average PSNR gain of 0.01 dB on Nam. These findings underscore the effectiveness of the proposed QFormer network structure in denoising real noisy images. QFormer leverages the sequential structure of QTB to refine global features, retrieve channel-related local features, and aggregate global-local features for capturing long-term dependencies. This further implies that QFormer adeptly utilizes these longrange (non-local) feature to enhance denoising performance.

To visually demonstrate the superiority of our method, Fig. 3 shows a visual comparison of different methods on different datasets for denoising real noisy images. We can observe that our QFormer achieves the best visual results in terms of noise removal and detail preservation. For example, neither Uformer nor Restormer can preserve the boundary contours of the fine pink regions, while the proposed QFormer can reconstruct and preserve the underlying edges. This further demonstrates that the correlation of color channel and longrange dependencies can improve denoising performance.

5.3 Ablation Study

To visually illustrate the impact of quaternion operation and identity mapping on image denoising, we use the following case to conduct experiments. Case.I means that only the convolution operator is used in the QFormer structure. Case.II means that only the primitive quaternion operator (i.e., without short skip connections) is used in the QFormer structure. Case.III means that only the proposed quaternion operator (i.e., using short skip connections) is used in the QFormer structure. It is worth noting that the convolution operator is an identity map of [Song et al., 2021].

Effect of Quaternion Operation As shown in Fig. 4, compared to QFormer (Case.I) consisting of pure convolution operators, QFormer (Case.III) with short skip connections retains and enhances more detailed information (e.g., boundary, edges). The experimental environment, experimental settings, and network structure of Case.I and Case.III are consistent. Therefore, the reason why the feature maps in Case.I and Case.III have large differences are that the quaternion operator and the vanilla convolution operator only model the correlation between the input noise image channels differently. This fully demonstrates that, compared with the vanilla convolution operator, the quaternion convolution operator can effectively and adequately model the color structure information of the input image to improve the feature representation ability. To quantify the impact of quaternion operation on image denoising performance, the PSNR/SSIM of Case.I, Case.II, and Case.III on real datasets are reported in Table 2. Compared with QFormer based on pure vanilla convolution operator (Case.I), QFormer with short skip connections (Case.III) improves PSNR by 1.09 dB, 1.51 dB and 0.65 dB on Nam, PolyU and SSID datasets. This illustrates that the quaternion operator can improve the image denoising performance by modeling the color structure information of the input image.

Effect of Identity Mapping In this section, as shown in Fig. 4, we perform a more intuitive visualization of feature maps within the proposed QFormer interiors on different settings to observe the differences. It can be seen from the figure that, compared with the QFormer without short skip connections (Case.II), high-frequency information (e.g., lines, points) is clearly emphasized in the feature maps generated by QFormer with short skip connections (Case.III). This illustrates that the lack of strict identity mapping of the primitive quaternion operator may be a major obstacle to the improvement of image denoising performance. At the same time, it also demonstrates that the short skip connection operation can overcome the defect of the non-strict identity mapping of the original quaternion operator by superimposing the input features on the output features. To quantify the impact of identity mapping on image denoising performance, the PSNR/SSIM of Case.I, Case.II, and Case.III on real datasets are reported in Table 2. From the table, it can be found that compared with QFormer without short skip connections (Case.II), QFormer with short skip connections(Case.III) improves PSNR by 1.22 dB, 1.58 dB and 0.67 dB on Nam, PolyU, and SSID datasets, respectively. By comparing the specific quantitative measurements, it can be observed that short skip connections can rise above the non-strict identity mapping of the primitive quaternion operator and greatly improve denoising performance.

5.4 Efficiency of QFormer

The reported results of the model sizes are shown in Fig. 5. It is worth noting that the proposed QFormer-B achieves the best image denoising performance on the SIDD, PonlyU, and Nam real noisy image datasets. In addition, on the SIDD dataset, compared with the state-of-the-art Restormer, the proposed QFormer-T, QFormer-S, and QFormer-B reduce the number of parameters by about 22.8 times, and 5.7 times and 3.0 times, respectively. Similarly, on the PonlyU dataset, compared with the state-of-the-art MIRNet, the parameters of the proposed QFormer-T, QFormer-S, and QFormer-B are reduced by about 36.0 times, 9.1 times and 4.8 times, respectively. On the Nam dataset, compared with the state-of-theart APD-Nets, the parameters of the proposed QFormer-T, QFormer-S, and QFormer-B are reduced by about 16.3 times and 4.1 times, and 2.2 times, respectively.

Additionally, we further investigate the advantages of quaternion operators, and QTB structure within QFormer. First, we discuss the effect of quaternions within QFormer, as shown in Table 3. We can find that once all the quaternion convolution and quaternion fully connected layers in QFormer are replaced by vanilla convolution and fully connected layers, the image denoising performance, and computational efficiency drop. Specifically, while keeping the overall structure of QFormer unchanged, compared with the model with quaternion operator (i.e., Case.d and Case.h), the image denoising performance PSNR/SSIM of the model without quaternion operator (i.e., Case.b and Case.f) decreases significantly on the three real noisy image datasets, with an average decrease of about 1.08 dB and 0.021, respectively. In comparisons in terms of efficiency, i.e., Case.j vs Case.l, Case.n vs Case.p and Case.r vs Case.t, the number of parameters, inference speed, and FLOPs all decline prominently. This fully demonstrates the great potential of the quaternion operator to achieve a better balance between the performance and efficiency of image denoising. The overall UNet structure is also used and unchanged, and the quaternion operator is introduced to replace the vanilla convolutions (i.e., Case.c, Case.j, Case.k, Case.o and Case.s). Compared with the UNet model using the convolution operator (i.e., Case.a, Case.e, Case.i, Case.m and Case.q), the image denoising performance and computational efficiency of the UNet with the quaternion operator are significantly improved. This further indicates that the quaternion operator can also play an important role in image denoising performance and efficiency without introducing the QTB structure.

Secondly, we discuss the effect of the QTB structure in QFormer. We compare UNet, which employs the quaternion operator (i.e., Case.c, Case.j, Case.k, Case.o and Case.s), with the proposed QFormer. From Table 3, it can be found that the PSNR/SSIM of QFormer with QTB structure on the three test sets of real noisy images is higher than those of Case.c and Case.j by about 1.51 dB and 0.030, respectively. In addition, QFormer with QTB structure improves the inference speed by about 70.46%. This suggests the proposed QTB structure is better at the ability to model the long-range dependencies for heavy noise removal with satisfactory efficiency.

6 Conclusion

In this paper, we propose a highly efficient Quaternion Transformer (QFormer) Network using Quaternion Transformer Block (QTB) for the image denoising task. The proposed QTB alternately captures both the long-range dependencies and local contextual features with sufficiently retrieving color structure information. In addition, from the mathematical perspective, we explained that quaternion operators are not strictly identically mapped in embedding deep neural networks for image denoising tasks, and thus proposed using short skip connections to ease this limitation. The proposed sequential-structured QTB is concise and can avoid considerable element-wise multiplications of computing selfattention matrices, thereby reducing computation complexity and improving practical inference speed. From extensive and comprehensive experiments on denoising tasks, the proposed QFormer achieves state-of-the-art results, demonstrating the satisfactory superiority of denoising and efficiency. We hope this pioneering efficient QFormer structure can encourage further exploration of the Quaternion Transformer architecture for image denoising tasks.

Acknowledgements

This work was supported in part by the NSFC fund (NO. 62176077, 62206073), in part by the Shenzhen Key Technical Project (NO. JSGG20220831092805009, JSGG20220831105603006, JSGG20201103153802006, KJZD20230923115117033), in part by the Guangdong International Science and Technology Cooperation Project (NO. 2023A0505050108), in part by the Shenzhen Fundamental Research Fund (NO. JCYJ20210324132210025), in part by the Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (NO. 2022B1212010005), and in part by the Guangdong Shenzhen joint Youth Fund under Grant 2021A151511074, in part by the Natural Science Foundation of Guangdong Province under Grant 2023A1515010893, in part by the Shenzhen Doctoral Initiation Technology Plan under Grant RCBS20221008093222010.