MetaISP: Efficient RAW-to-sRGB Mappings with Merely 1M Parameters

1 Introduction

Over the past decade, DSLR cameras have increasingly been supplanted in various application scenarios due to their lack of portability. This shift has intensified the interest in achieving high-quality sRGB images without relying on large sensors and lenses. As sensor size constraints reach their maximum, researchers have turned their attention to improving the image signal processing (ISP) pipeline, which aims to reconstruct high-quality sRGB images from raw sensor images [Ignatov et al., 2022a]. The conventional ISP pipeline consists of multiple manually designed modules. Unfortunately, the accumulation of errors from each processing module gradually degrades the overall reconstruction quality of the sRGB

∗Corresponding author.

image [Liang et al., 2021b].

To address this issue of accumulated reconstruction errors, the concept of an end-to-end deep ISP model has been introduced in the literature [Schwartz et al., 2018], which integrates the entire ISP pipeline into one single unified learnable model. Despite the encouraging performance in RAWto-sRGB tasks, these existing end-to-end deep ISP models present two distinctive challenges:

• (1) Diverse Conditions. The diversity in environmental conditions and camera settings under which images are captured makes it challenging for the model to establish accurate mappings for every image pair;
• (2) High Efficiency Requirement. Since the deep ISP model is commonly deployed on resource-constrained devices, prioritizing efficiency and lightweight design becomes crucial to ensure its practical value.

To attain remarkable reconstruction quality under diverse conditions, certain heavy ISP models [Ignatov et al., 2020a; Ignatov et al., 2020b; Kim et al., 2020; Dai et al., 2020; Liang et al., 2021b; Zhang et al., 2021] have been proposed. Yet, these models all come with complex structures and a substantial parameter count. To be suitable for resource-constrained devices, some lightweight ISP models [Raimundo et al., 2022; Ignatov et al., 2023b; Hsyu et al., 2021; Ignatov et al., 2023a; Ignatov et al., 2022a] have also been introduced. Unfortunately, these lightweight models all inevitably compromise the quality of the reconstructed sRGB image. As such, it remains a challenging problem in the literature to generate sRGB images of comparable or superior quality to those produced by heavy models, while utilizing significantly fewer parameters and FLOPs like those lightweight models.

In this paper, we introduce MetaISP, an adaptive network for RAW to sRGB mappings that offers a lightweight architecture and efficient inference while consistently producing high-quality outputs. Our rationale comes from the substantial variations of the mapping relationship between RAW and sRGB images under diverse conditions. Motivated by this observation, we propose an elaborated sample-wise dynamic network that adaptively adjusts its parameters and architecture in response to diverse inputs. Unlike existing models with fixed parameters, our method allows for automatic and flexible learning of accurate RAW to sRGB mappings under diverse conditions. Consequently, it presents significant facilitators to the model training process and enhances the model's ability to generalize. As depicted in Figure 1, our approach achieves high-quality sRGB images while significantly reducing the number of parameters and required FLOPs.

The key element of our MetaISP is a U-Net-like architecture comprising two novel blocks: the meta channel correction block (MCCB) and the meta spatial correction block (MSCB). MCCB obtains stronger parameter prediction ability and improves the learning capability for the relationship between feature map channels by predicting the contentdependent weights of the basis matrices. While MSCB innovatively calculates the cosine similarity between each pixel in the feature map and the adaptive correction vector to get the spatial attention map efficiently and achieve stronger representation ability. Additionally, the meta shallow feature extraction block (MFEB) and the meta reconstruction block (MRB) are also proposed to further improve the quality of the sRGB image. MFEB realizes flexible sample-wise feature extraction by predicting dynamic weights for five predefined convolution kernels based on the input RAW image. Additionally, MRB enhances global alterations by utilizing the output from U-Net's bottom layer to guide the final image generation process.

Our contribution is a novel deep ISP model that achieves high-quality sRGB images while significantly reducing computational complexity. This is achieved by four complementary and lightweight dynamic blocks entitled MCCB, MSCB, MFEB, and MRB, which adaptively adjust model parameters and structure based on input images. As shown in Figure 1, extensive experimental results on two large-scale datasets demonstrate the superiority of our MetaISP: 23.80dB PSNR on ZRR [Ignatov et al., 2020b], exceeding the previous SOTA 0.19dB with only 9.2% of its parameter count and 10.6% of its FLOPs; 25.06dB PSNR on MAI21 [Ignatov et al., 2022b], exceeding the previous SOTA 0.17dB with only 0.9% of its parameter count and 2.7% of its FLOPs.

2 RelatedWork

Recent approaches in Image Signal Processing (ISP) aim to enhance image quality by leveraging deep learning to transform Bayer RAW format images into high-quality RGB images. These methods formulate ISP tasks as end-to-end processes, replacing the traditional sequential modules with a single model.

Schwartz et al.[Schwartz et al., 2018] pioneered this approach by introducing DeepISP, which utilizes Convolutional Neural Networks (CNNs) to create a fully integrated ISP pipeline. Building upon this, Ignatov et al. [Ignatov et al., 2020b] developed Pynet, a network designed around the pyramid architecture, thereby further advancing the receptive field. This development encouraged many researchers to adapt Unet-based models for this task. Zhang et al. [Ignatov et al., 2020a] proposed MW-ISPNet, a model that first uses a wavelet transform to replace the downsampling and upsampling stages in the deep ISP model. Additionally, Dai et al. introduced AWNet [Dai et al., 2020], incorporating wavelet transform and global context attention to enhance image quality by providing the model with a larger receptive field. Raimundo et al. [Raimundo et al., 2022] proposed LAN, a model integrating spatial attention to capture spatial information within images. Liang et al.[Liang et al., 2021b] suggested CamerNet, arguing that ISP should be split into two relatively uncorrelated segments. They deployed two separate U-Net models to handle restoration and enhancement, respectively. LiteISP [Zhang et al., 2021] achieves better results with a lighter model by solving the problem of misalignment between RAW and sRGB images in the dataset.

In an effort to boost ISP performance on mobile platforms, [Ignatov et al., 2022a] proposed PynetV2, a lightweight and efficient model that expands upon the concept of channel attention. [Hsyu et al., 2021] proposed CSANet, which employs a dual attention module that leverages both channel and spatial attention. [Ignatov et al., 2023a] further innovated with the proposal of MicroISP, which operates at the original scale, thereby reducing the GPU memory requirements. This approach enables the processing of larger images and ensures faster operational speed.

These state-of-art ISP algorithms still suffer from one major flaw: in order to accommodate the diversity of input RAW images derived from real-world environments, these methods often resort to expanding the model's capacity to enhance its generalizability. However, this invariably leads to an escalation in computational cost. In contrast to existing methods, our approach eliminates the dependency on high-cost models, while still achieving high-quality generation of RGB images.

3 ProposedMethod

Our primary goal is to develop a deep ISP model for highquality sRGB image restoration from camera sensor outputs, with reduced parameters and FLOPs. To achieve exceptional performance with low computational complexity, we propose MetaISP, which utilizes a dynamic network concept. In this section, we introduce MetaISP's overall architecture and four key components: (a) meta channel correction block (MCCB), (b) meta spatial correction block (MSCB), (c) meta shallow feature extraction block (MFEB), and (d) meta reconstruction block (MRB).

3.1 Overview

As illustrated in Figure 2, the overall architecture of MetaISP follows a U-shaped encode-decoder structure. The U-net architecture enables the model to capture both low-level and high-level image features, enabling robust representation learning. Given a 4-channel RAW image Iraw ∈ R4×H/2×W/2 , MetaISP first employ MFEB to adaptively extract shallow feature F0 ∈ RC×H/2×W/2 , where H/2×W/2 denotes the spatial resolution and C is the number of channels. Then the shallow features F0 are fed into a 4-level Unet for multi-scale deep feature extraction and reconstruction. Each level of the U-net contains only two correction blocks which are composed of proposed MCCB and MSCB. From top to bottom levels, the number of feature map channels are {C, C, C, 2C}. The model benefits from a comparatively smaller number of channels and blocks, resulting in a significant reduction in computational complexity. The downsampling and upsampling of the feature maps are achieved using discrete wavelet transformation [Liu et al., 2018] and a 3×3 convolution layer. The residual connections are implemented through per-pixel addition. Finally, the feature map of the bottom level F2 ∈ R2C×H/16×W/16 and the output of the Unet F1 ∈ RC×H/2×W/2 are fed into MRB to reconstruct the sRGB image Irgb ∈ R3×H×W .

3.2 Meta Channel Correction Block (MCCB)

Channel attention [Hu et al., 2018] is a simple dynamic parameter block that is commonly used in many low-level vision tasks including deep ISP to capture channel-wise dependencies and highlight informative features while suppressing irrelevant or redundant ones. By incorporating the channel attention mechanism into CNN architectures, models can adaptively assign different importance levels to channels and dynamically adjust their contributions during feature extraction. It realizes computational efficiency and brings global information to the feature map. However, its linear transformation through simple channel-wise multiplication with attention weights M ∈ RC×1 exhibits limited fitting capacity, while its simplistic attention prediction layers may lead to suboptimal attention weights. Consequently, it becomes necessary to continually increase the depth and width of the channel attention block to compensate for these drawbacks. To further improve the ability to learn inter-channel relationships, Restormer [Zamir et al., 2022] proposed a more powerful and dynamic method based on the self-attention (SA) mechanism [Vaswani et al., 2017]. It calculates the response of a specific channel by performing a weighted sum across all other channels, thereby generating an attention map M ∈ RC×C . Nevertheless, generating the attention map involves computing cross-covariance across channels, which is computationally more expensive compared to CNN-based channel attention.

To achieve the high performance of self-attention while maintaining low computational complexity similar to CNN, we propose MCCB. The architecture of MCCB is shown in Figure 2. The block begins with layer normalization [Ba et al., 2016], which enhances the stability of the training process and enables the utilization of larger learning rates. The following 3×3 convolution and GELU [Hendrycks and Gimpel, 2016] aggregate the local spatial context. Then the feature map X ∈ RC×H×W is projected to a high-dimensional space Y ∈ RK×H×W to obtain richer features. Unlike MDTA and channel attention, which directly calculate the weights of each channel, MCCB joint learns several basis matrices {ϕn}n=1,...,N together with a simple prediction branch p which predicts weights {wn}n=1,...,N = p(X) for each basis matrices. For an input feature map X ∈ RC×H×W , the adaptive correction matrix M ∈ RK×K is obtained as:

Next, we reshape the high-dimensional feature map and perform matrix multiplication with the input-adaptive correction matrix, similar to the self-attention mechanism. The transformation process is defined as:

where ̂R ∈ RHW×K and R ∈ RHW×K are the input and output reshaped feature maps, M ∈ RK×K is the input-adaptive correction matrix, ∗ stands for matrix multiplication. Finally, we project the transformed feature map ̂ Y ∈ RK×H×W back to its original dimension ̂X ∈ RC×H×W and establish a residual connection [He et al., 2016] with the input X ∈ RC×H×W .

Our MCCB utilizes only a few basis matrices to transform the feature maps and employs dynamic soft weights to achieve content-adaptive channel transformation. This mechanism enables us to leverage the same low computational complexity as channel attention while effectively learning complex inter-channel relationships, similar to the transformer model.

3.3 Meta Spatial Correction Block (MSCB)

In addition to channel attention, spatial attention is another dynamic parameter mechanism that focuses on capturing and emphasizing relevant spatial regions in an image. One common approach in spatial attention is to utilize convolutional neural networks (CNNs) to learn attention maps that highlight relevant spatial regions. These attention maps are then used to modulate the feature representations in subsequent layers, enabling the network to focus on discriminative regions while suppressing irrelevant or noisy information. The CNN-based spatial attention leverages parameter sharing across spatial locations, enabling efficient processing and reducing computational complexity. However, it has a limited receptive field, thus preventing it from modeling long-range pixel interactions. While the self-attentionbased spatial attention [Chen et al., 2021; Liang et al., 2021a; Wang et al., 2022] has exceptional capabilities in global context modeling but suffers from high computational complexity. The complexity of the attention block is quadratic in relation to the input size, making it impractical to apply to highresolution images in ISP tasks. To simultaneously achieve the low computational complexity of CNN and the global context modeling capabilities of self-attention in spatial attention, we propose the MSCB.

As is shown in Figure 2, MSCB shares a comparable overall architecture with MCCB. We simultaneously learn multiple basis vectors {εn}n=1,...,N along with a straightforward prediction branch, denoted as p, which predicts dynamic weights {wn}n=1,...,N = p(X) for each basis vectors. The adaptive correction vector V ∈ R1×K is obtained through weighted addition of multiple basis vectors:

We assign attention weights to each pixel in the feature map by computing the cosine similarity between each pixel and the correction vector. The transformation process is defined as:

where ̂ Y ∈ RK×H×W and Y ∈ RK×H×W are the input and output high-dimensional feature maps, V ∈ R1×K is the sample-wise adaptive correction vector, S ∈ R1×H×W is the spatial attention map, ⊗ stands for pixel-wise multiplication and ∗ stands for dot product. To normalize the attention weights, we compute the cosine similarity, ensuring that all values fall within the range of 0 to 1. This normalization process helps to achieve smoother training and reduce the excessive influence of attention weights on a few pixels. Pixels that exhibit higher similarity to the correction vector will be assigned higher attention weights, while pixels that show less similarity to correction vector will be assigned lower attention weights. MSCB achieves linear complexity O(HW) while effectively utilizing the global information of the feature map.

The visualization of attention maps in MSCB is depicted in Figure 3, where the model leverages the attention mechanism to discern intrinsic spatial correlations. Attention maps at different levels reveal varying inherent relationships.

3.4 Meta Feature Extraction Block (MFEB)

Considering the varied demands for high-frequency (detailfocused) and low-frequency (color and illumination) information restoration across images, we advocate for a dynamic, image-specific feature extraction. MFEB is chiefly partitioned into three components. We first implement a 3×3 standard convolution for baseline performance. Next, to gather global information, we apply pre-defined kernel functions such as the box filter and Gaussian filter. Lastly, we use Sobel-X, Sobel-Y, and Laplacian kernel functions for local feature extraction. A prediction network is also incorporated to generate weights for each branch based on the input image. The predicted weights {Wn}n=1,...,6 indicate the significance of each feature extraction branch, and {Fn}n=1,...,6 denote features extracted from each branch. The formula below depicts the weighted sum for each instance.

3.5 Meta Reconstruction Block (MRB)

The crux of Learnable ISP tasks lies in color mapping. Modern image restoration typically employs U-Net-shaped networks to extract and decode multi-scale information for image restoration. Still, aspects like lighting and color require sequential decoding, typically at the U-Net's lowest layer. To address this, we introduce the meta reconstruction block. As shown in Figure 4, MRB uses the output from the U-Net's base layer to guide final image reconstruction.

In the preceding equation, Xh and Xl depict the outputs from the top and bottom layers of the U-Net, respectively. Fr signifies a series of 3 × 3 convolutions with pixel shuffling, and Fchannel comprises standard 1 × 1 convolutions and a global average pooling, yielding a C × 1 × 1 output that captures inter-channel correlations. Finally, ⊗ indicates channel-wise multiplication.

4 Experiments

4.1 Implementation Details

Datasets. We conduct experiments on two publicly available RAW to sRGB datasets: the Zurich RAW to RGB (ZRR) dataset [Ignatov et al., 2020b] and the MAI2021 dataset [Ignatov et al., 2022b].

ZRR dataset is a large-scale dataset consisting of 48k RAW-sRGB pairs of size 448 × 448. We follow the official division that 46.8k are used for training and 1.2k are used for testing. To further address the misalignment issue in the ZRR dataset, we utilized a global color mapping (GCM) module [Zhang et al., 2021] in combination with a pre-trained PWCNet [Sun et al., 2018] to warp the sRGB images with the corresponding RAW images. In contrast to the joint training strategy of [Zhang et al., 2021], we pre-train a GCM module and utilize its output to align the sRGB images before training the ISP model for fairness and certainty in our comparisons.

We also used the MAI21 dataset which consists of 24k RAW-sRGB pairs of size 256 × 256. The dataset was then randomly divided into two parts, with 23k samples allocated for training and 1k samples reserved for testing.

Training Details. Our model is implemented in PyTorch [Paszke et al., 2019] and trained on 4 Nvidia Titan X GPUs with a batch size of 32. The parameters of the network are optimized with ADAM [Kingma and Ba, 2014] algorithm.

For the ZRR dataset, all the training images were augmented by random horizontal and vertical flipping during the training. Our MetaISP consists of a four-level U-Net architecture, with channel numbers of 32, 32, 32, and 64 from top to bottom. The model undergoes a two-stage training process. First, the model is trained for 80 epochs with an initial learning rate of 5e-4 which is decayed to half after 50 epochs. The loss function is a combination of VGG-based perceptual loss [Johnson et al., 2016], SSIM loss [Wang et al., 2004] and Charbonier loss [Zhang et al., 2018]:

Next, the model is fine-tuned for an additional 5 epochs with a learning rate of 2e-5 . Only MSE loss and SSIM loss are employed for final tone adjustments and edge rendering:

For the MAI21 dataset, no data augmentation methods were employed throughout the training phase. We further reduce the depth and width of MetaISP to make it more lightweight. The training process is similar to the ZRR dataset.

4.2 Experimental Results

ZRR Dataset. Our proposed model, MetaISP, was benchmarked against four cutting-edge models on the ZRR dataset, demonstrating superior performance in all metrics, including PSNR, SSIM and ∆E (see Table 1). Notably, the MetaISP's excellence was achieved with significantly fewer parameters and computational resources, requiring only approximately 8.6% of the parameters and 9.5% of the Flops utilized by LiteISP [Zhang et al., 2021]. Further corroborating its superiority, MetaISP synthesizes output images of an exceptional quality that are rich in detail and exhibit superb restoration of global information, such as illumination and color. This qualitative superiority is clearly illustrated in Figure 5.

In Table 2, we provide a detailed comparison of the actual memory requirements, running latency, and frames per second (FPS) across various image resolutions to offer a comprehensive evaluation of efficiency. Our model demonstrates significant advantages in efficiency over the SOTA

LiteISP model, particularly in processing high-resolution images. Additionally, the analysis reveals that the computational complexity of the MetaISP model is linearly correlated with the input size. These efficiency assessments were conducted using a single Titan X GPU.

MAI21 Dataset. An extensive evaluation on the MAI21 dataset affirmed MetaISP's superior performance and computational efficiency compared to state-of-the-art models (see Table 1). Remarkably, MetaISP outshone models like MicroISP and PyNet-V2 in the majority of evaluation metrics, despite these models leveraging additional data. The formidable efficiency of MetaISP becomes evident when considering that it utilizes a scant 1.1% of LiteISP's parameters and 2.4% of its computational burden (in FLOPs), while still managing to outstrip LiteISP in performance. In terms of image quality, sharpness, and color restoration, MetaISP's output distinctly excels, a fact clearly illustrated in Figure 6. It is particularly noteworthy that our model significantly outpaces other models in the aspect of color information restoration.

4.3 Ablation Studies

In this section, we conduct extensive experiments to measure the contributions of our proposed blocks. Experiments are performed on the aligned ZRR dataset, and models are trained on image patches of 448×448 for 85 epochs.

Effectiveness of the Four Proposed Blocks

To evaluate each individual component in MetaISP, we conducted additional experiments as shown in Table 3. The baseline is the MetaISP trained with all four components. When replacing MFEB with a commonly used 3×3 convolution layer, the PSNR decreases by 0.11 dB. The inclusion of MRB helps mitigate the overfitting issue in the model, as evidenced by a PSNR drop of 0.29 dB when removing it. In addition to enhancing the quality of output images, MFEB and MRB contribute only marginally to the computational complexity of the model. MCCB and MSCB serve as fundamental components within the U-shaped encoder-decoder architecture of MetaISP. Removing MCCB results in a PSNR drop of approximately 0.5 dB, while removing MSCB leads to a larger drop of 0.63 dB.

Various Numbers of Basis Matrices and Vectors

To determine the number N of basis matrices and vectors, we evaluate the performance of our model by setting N = {4, 8, 10, 16, 32} as shown in Table 3. We have chosen to set N as 10 in order to strike a balance between efficiency and performance in our model. Further increasing the value of N does not result in significant performance improvements and may potentially lead to overfitting issues.

Design of MFEB

The results presented in Table 3 demonstrate that the removal of global feature extraction kernels or local feature extraction kernels degrades the performance of the baseline model. This observation suggests that both the global branches and local branches within MFEB contribute to the flexible shallow feature extraction process, thereby enhancing the overall performance of our model. Additionally, the utilization of these pre-defined kernels has a negligible impact on the number of parameters and FLOPs of the model.

Superiority to other existing blocks

For further comparative analysis, we substituted our proposed MSCB and MCCB with the widely-utilized CNNbased RCAB [Hu et al., 2018] and the Self-attention-based MDTA [Zamir et al., 2022]. As depicted in Table 4, our blocks demonstrate comparable efficiency to CNN-based approaches while outperforming the self-attention mechanism in terms of overall performance.

5 Conclusion

In this paper, we propose a novel learnable ISP model called MetaISP. Our method aims to learn accurate mappings between RAW and sRGB under diverse environmental conditions, while significantly reducing computational complexity. This objective is accomplished by employing adaptive model parameters and architecture adjustments according to the characteristics of input images. Experiments on two large-scale public datasets demonstrate that our model outperforms the state-of-the-art methods while having substantially fewer parameters and FLOPs. Extensive ablation experiments provide compelling evidence for the effectiveness of the four main components.

Ethical Statement

This research strictly follows the highest ethical standards in image restoration. We responsibly and ethically utilized data sources, including images and algorithms, throughout the study. We have adhered to institutional and governmental guidelines for ethical research practices, ensuring transparency and reproducibility in our methodology. The research does not involve studies with human participants or animals conducted by any of the authors. In cases where external contributions were incorporated, proper credit was given to maintain academic integrity and uphold scientific community standards.

Acknowledgments

This research is supported by the Singapore Ministry of Education Academic Research Fund Tier 1 (WBS: A000944001-00).

Contribution Statement

Zigeng Chen and Chaowei Liu contributed equally to this work.