KPN-MFI: A Kernel Prediction Network with Multi-frame Interaction for Video Inverse Tone Mapping

1 Introduction

High dynamic range is the most pervasive feature of the modern televisions (TVs). The effect of the HDR video rendered on the HDR display device is clearly approximating the human eye seeing the real world. Nevertheless, the majority of the transmitted visual contents are still in SDR format, e.g., the Digital TV and Internet Protocol TV (IPTV). Therefore, there is an urgent demand to convert the legacy SDR contents to their HDR version.

∗Corresponding author

Transforming SDR contents into HDR ones is frequently referred to as inverse tone mapping (iTM). The problem of estimating HDR images from SDR photographs has been studied for many years [Liu et al., 2020; Santos et al., 2020; Cao et al., 2021]. However, the video-based iTM task is challenging due to the video data always containing more complex scene contexts, motion, etc., in comparison with image data. It is difficult to model both spatial and temporal information simultaneously for video sequence data. As a result, some existing video-based iTM methods only focus on enhancing the quality of a single frame, ignoring the temporal context information between consecutive frames. In general, for video data, inter-frame temporal consistency is even more important than intra-frame visual quality, to a certain extent. However, these existing video-based iTMmethods which are only based on single frame processing, cannot capture available information conveyed by the context information in the neighboring frames, resulting in compromised quality of results and cannot control the temporal consistency between frames.

To overcome this challenge, we propose a video iTM method taking advantage of both spatial and temporal information. Different from these previous image-based iTM methods which aim to predict the relative luminance of the scenes in the linear domain, our model directly predicts HDR videos in the HDR display format in the pixel domain, including the color gamut must be expanded from BT.709 [ITU-R, 2015a] to BT.2020 [ITU-R, 2015b], the bit-depth increased from 8 bit/pixel to 10 bit/pixel, and the optical-electro transfer function (OETF) also changes from gamma [ITU-R, 2011] to Perceptual Quantizer (PQ) [ITU-R, 2014]. To this end, we propose to model spatial-temporal information simultaneously in our video iTM method by integrating the multiframe interaction (MFI) module into an encoder-decoder kernel prediction network (KPN). Firstly, a kernel prediction network framework is adopted in order to preserve the quality of the single frame video frames and improve the efficiency of the model by using a parameter sharing mechanism. Secondly, the multi-frame interaction module is designed to capture short-term spatial-temporal dependence between adjacent video frames. With the feature between adjacent frames exploited, not only the consistency between frames can be maintained, but also the quality of a single frame is improved. Additionally, any image-based method can capture the spatial and temporal features of a video sequence by introducing the MFI module, so that it can be readily extended to a video iTM method. Furthermore, a brightness consistency loss is developed to optimize our model to focus more on the brightness consistency between adjacent frame video frames.

Our contributions are summarized as follows:

• A multi-frame interaction module is introduced for modeling spatial-temporal context information, which aims at the feature interaction between frames through leveraging feature re-calibration based on the spatialtemporal relation between consecutive frames.
• A kernel prediction network framework is proposed not only to ensure the intra-frame iTM ability of the encoder-decoder model for single frames, but also to keep the model efficient by using the parameters sharing mechanism.
• A brightness consistency loss function based on the Gaussian pyramid is proposed in order to preserve the brightness consistency between adjacent video frames.

2 RelatedWork

2.1 Image-based iTM

With the rapid development of Convolutional Neural Networks (CNNs) in many computer vision tasks, (e.g., superresolution [Zhang et al., 2021] and denoise [Lin et al., 2021]), CNN-based iTMmethods have been developed. For instance, Liu et al., [Liu et al., 2020] use multiple sub-networks to model the reverse pipeline of the camera producing SDR images from HDR. [Santos et al., 2020] reconstruct HDR images by using a CNN with masked features and perceptual loss. HDRUNet [Chen et al., 2021b] predicts a non-linear 16-bit HDR image with three sub-networks. Cao et al., [Cao et al., 2021] adopts pixel-wise kernel convolution of the input SDR image to generate the HDR results to avoid some artifacts in the under/over-exposed regions.

However, these image-based iTM methods usually aim to predict the relative luminance of a scene in the linear domain. Thereby, these image based methods can not directly output display format data and post-processing (e.g., gamut mapping, EOTF, quantization) must be performed.

2.2 Video-based iTM

In general, extant deep learning based iTM literature primarily focuses on HDR image reconstruction. Very few studies [Chen et al., 2021c; Kim et al., 2019; Kim et al., 2020]

have been proposed to address the video iTM tasks. One of the latest video iTM methods HDRTVNet [Chen et al., 2021c] use three sub-networks to simulate the key steps in the HDR video generation process. [Kim et al., 2020; Kim et al., 2019] divided the input SDR frame into base and detail layers and then reconstruct the detail and enhancement local contrast, respectively. Nevertheless, all the above approaches can be seen as single-frame iTM approaches, as they do not use any advantageous information available in the neighboring frames. As a result, these single-frame methods may cause temporal inconsistency and has limited performance in single-frame.

To solve these problems of existing video iTMmethods, in this paper, we design a lightweight model, in which the parameters are shared in the encoder and decoder model for adjacent frames. With the assistance of the multi-frame interaction module, our model exploits the spatial-temporal context information to improve single frame reconstruction quality and reduce temporal inconsistency between adjacent frames.

3 Proposed Approach

It is well known that, in video-based reconstruction tasks, we need to consider both the spatial structure within a single frame and the temporal consistency between adjacent frames. However, existing video-based iTM methods ignore the inter-frame information. Considering both intraframe reconstruction ability and inter-frame brightness coherency, we design a kernel prediction network with multiframe interaction, named KPN-MFI, which comprises an encoder-decoder backbone, multi-frame interaction module, and adaptive pixel-wise convolution module. Concretely, a basic encoder-decoder with an adaptive pixel-wise convolution module is adopted to guarantee the mapping quality within each frame. Moreover, the MFI module is designed to captures the inter-frame spatial-temporal context information and preserve consistency by exploiting the correction between adjacent frames. The detail of the proposed KPN-MFI is shown in Fig. 1. The KPN-MFI is organized under a parameter sharing mechanism. It means that adjacent frames share the same parameters of the backbone network.

3.1 Model Overview

As shown in Fig. 1, given three adjacent SDR frames (St-1, St, St+1), the KPN-MFI produces three HDR frames (Ht-1,Ht,Ht+1) simultaneously by passing them to the same encoder-decoder model. Specifically, for each frame of a sequential (St-1, St, St+1), multilevel features are extracted by the encoder. The feature extraction procedure can be written as:

where E(·) denotes encoder model, Xli and Xhi denote the extracted low-level and high-level feature maps, respectively.

In the encoder, low-level features Xli are with high resolution and sensitive to the local variations. It means they would be beneficial to recover fine-grained information. Thus, as shown in Fig. 1, we include a long skip connection in the backbone to pass Xli to the decoder for a fine intra-frame reconstruction. In contrast, high-level features are with larger receptive fields and thus more robust to feature displacements. Since feature displacements commonly appear between adjacent frames, we would like to apply the MFI modules to the high-level features. In this way, we can well capture the spatial-temporal context information without suffering from the alignment problem caused by moving objects in videos. Before the MFI modules, we first include selfattention (SA) modules [Zhang et al., 2019b] to map the high-level features Xhi to re-calibrated features denoted as Yi. Subsequently, the re-calibrated feature maps of sequential frames are further fed into the MFI module and output interacted feature maps. It can be expressed as:

where C(t-1,t+1)→t denotes the interacted feature of the frame St obtained from the intra-frame features Yt-1 and Yt+1. MFI(·) is the MFI module, which will be described in Section 3.2.

In the decoder, a set of adaptive kernels is produced from the interacted features and low-level features, which is formulated as Kt = D(C(t-1,t+1)→t,Xlt ), where D(·) is decoder model and Kt ∈ Rc×k2×h×w is a tensor consisting of pixel-wise kernels. The output HDR frame is produced by a pixel-wise convolution (PC) operation on the input SDR frame, as shown in the purple block in Fig. 1. It has been demonstrated that potential artifacts can be effectively alleviated by predicting pixel-wise kernels rather than HDR pixel [Cao et al., 2021]. The PC operation is in the form of

where Hc,i t ∈ R is the output HDR frame H at ith pixel and cth channel. Ω(i) denotes the k × k convolution window around ith pixel. Kc,i ∈ Rk×k denoting the kernel at ith pixel and cth channel. pi denotes 2D pixel coordinates and [pi - pj ] ∈ {(- k-1 2 ,- k-1 2 ), (- k-1 2 ,- k-1 2 +1), ..., ( k-1 2 , k-1 2 )} denotes the position offset between ith and j th pixels.

It is worth noting that if we exclude the MFI modules in Fig. 1, it is essentially a deep architecture designed for image iTM. The MFI module is proposed to encourage interframe interactions and capture spatial-temporal context information. Consequently, the developed MFI module can be embedded in any other image iTM architecture to achieve a successful video iTM, as demonstrated in Section 3.2.

3.2 Multi-Frame InteractionModule

Inspired by the non-local block in [Wang et al., 2018] capturing long-range dependencies, a multi-frame interaction (MFI) module is proposed to fully utilize the spatial-temporal context information. Different from the original non-local block, the MFI pays attention to the relationship between adjacent frames (St-1, St+1) and the current frame St. Intuitively, adjacent frame information can compensate for the current frame information reconstruction in both spatial and channel domains, which motivates us to highlight the relationship between multiple frames in spatial and channel domains. Therefore, the spatial and channel interaction operation (i.e., cosine distance and channel attention) between adjacent frames is introduced to make the model able to capture the spatialtemporal context information. Specifically, as shown in the orange block in Fig. 1, we first use 1 × 1 convolution layer to extract the features P ∈ Rc×h×w from the re-calibrated high-level feature Yt-1, and (Q,V) ∈ Rc×h×w from Yt. Then, we reshape (P,Q) to Rc×m and V to Rm×c , i.e., P = [p1 , p2 , ..., pm], Q = [q1 , q2 , ..., qm], and V = [v1, v2, ..., vc], where m = h × w is the number of spatial positions on each feature map. (pi , qi ) ∈ Rc and vi ∈ Rm are the feature vectors at the ith spatial position in P, Q, and V respectively.

The relevance between P and Q is computed with cosine distance and get the correlation map R ∈ Rm×m:

The interaction relationship between frame St-1 and frame St is measured as:

where Z(t-1)→t ∈ Rm×c , ri is ith row of R, and vj is j th column of V. The feature Z(t-1)→t is reshaped to Rc×h×w . Likewise, Z(t+1)→t ∈ Rc×h×w can be calculated in the same manner.

Then, the reshaped interaction relationship features Z(t-1)→t and Z(t+1)→t are fusion using 1 × 1 convolution layer and obtain multi-frame interaction relationship map Zt , which can be regarded as spatial attention in multi-frame context information. In order to further preserve the channel attention between Zt and Yt, a channel attention module is adopted as function:

where ∗ presents hadamard product, GAP and FC denote global average pooling operation and fully connection operation, respectively. Mt ∈ Rc×h×w , ω ∈ Rc×1×1 is employed to learn the channel-wise weights for Yt in terms of preserving global properties.

At last, we use a residual attention mechanism, where the initial feature maps Yt are element-wisely weighted by (1 + Mt ) for preserving feature consistency and compensating information. Thus, the multi-frame interaction feature map with spatial-temporal information can be formulated as:

In this way, we capture inter-frame correlation according to the spatial-temporal pixel-wise relation between multiple sequential frames. Similarity, we can get the multi-frame interaction feature map C(t,t+1)→t-1 or C(t-1,t)→t+1.

3.3 Loss Function

Our loss function contains an intra-frame content loss LC and an inter-frame brightness consistency loss LT . Specifically, the intra-frame content loss function is formed as:

where H′ denotes the ground truth HDR frame, Tanh(·) function is adopted to balance the low and high luminance values impact.

In order to ensure the temporal consistency further, we propose an inter-frame brightness consistency loss LT . Some existing inter-frame consistency loss uses the optical flow to align the frame, e.g., temporal stability loss in [Lai et al., 2018] and Long-term temporal loss in [Zhang et al., 2019a]. However, the current best inter-frame alignment methods (e.g., [Hui and Loy, 2020] and [Chen et al., 2021a]) have a limited effect on complex scenes. If we calculate the consistency loss between unaligned frames, it will reduce the model performance. To alleviate the impact of misalignment, we propose decomposing the predicted frame with a Gaussian pyramid. With the decomposition process of the Gaussian pyramid, the misaligned high-frequency information between the misaligned frames is gradually eliminated. Concretely, we first align the predicted frame Ht-1 and Ht+1 to Ht (defined as H(t-1)→t and H(t+1)→t) by an optical flow model Liteflownet3 [Hui and Loy, 2020]. Then, we obtain the illumination component (B(t-1)→t, B(t+1)→t ) of aligned frames (H(t-1)→t, H(t+1)→t) according to the classic Retinex theory [Land and McCann, 1971]. Finally, the illumination component is decomposed in a Gaussian pyramid, and the loss is calculated for each layer of the pyramid composition. The inter-frame brightness consistency loss LT is defined as:

where j means the number of pyramid decomposition layers and l is the total number of the pyramid decomposition layers (we set l = 5 in our experiments), Bj is j th illumination component, we use bilateral filter (with parameter d = 9, sigmaColor = 10, sigmaSpace = 5) to filter the frame to approximate the illuminance image in logarithm domain. γj is coefficients to measure the importance of the low-frequency pyramid component. During the pyramid decomposition process, the component with the smaller resolution carries the less high-frequency detail information and the more accurate representation of low-frequency information (such as global brightness ), and the less misaligned information between frames. Therefore, we set [γ1, γ2, γ3, γ4, γ5] = [1/16, 1/8, 1/4, 1/2, 1]. The Gaussian function with (ksize = 9, σ = 1.7).

The overall loss function to optimize the model is given by:

where α is a constant to balance LC and LT (α = 0.05 in our implementation).

4 Experiments

4.1 Experimental Settings

Dataset. We collect 21 pairs of 4K-UHD HDR videos under HDR10 standard and their SDR counterpart from YouTube, as in [Chen et al., 2021c; Kim et al., 2020]. 17 video pairs are used for training and the left 4 videos for testing. We sample three consecutive frames every two seconds of each video, and then we delete the frames with scene transitions. Thus, 3,241 pairs are generated for training while 482 pairs for testing are generated. The frames are cropped to 1024 × 1024 with overlapping, and 42, 608 pairs are generated totally for training.

Evaluation Metrics. In our experiments, we assess the performance in terms of the Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM) [Wang et al., 2004], ∆EITP [ITU-R, 2019] and HDR-VDP3 [Mantiuk et al.,

2011]. ∆EITP and HDR-VDP3 are designed for HDR videos. Furthermore, we choose the MADB in [Jiang and Zheng, 2019] to validate the temporal consistency of models.

4.2 Experimental Results

Qualitative Evaluations

To evaluate the performance of the proposed approach, we compare our approach against several existing state-of-the-art video-based iTMmethods, including SRiTM (ICCV) [Kim et al., 2019], JSINet (AAAI) [Kim et al., 2020] and HDRTVNet (ICCV) [Chen et al., 2021c]. Since there are few video-based iTM methods, we also compare our approach with several existing image-based methods, including HuoPhysEO [Huo et al., 2014], DeepHDR (TOG) [Santos et al., 2020], KPNiTM [Cao et al., 2021], and HDRUnet (CVPR) [Chen et al., 2021b]. All these methods are retrained on our dataset.

As shown in Table 1 and Fig. 2, our method KPN-MFI outperforms other methods by a large margin on PSNR, ∆EITP , and HDR-VDP3, but a slight decrease on SSIM. Quantitative results show the effectiveness of the proposed method. Besides, the calculation cost and parameters are compared. Our model can achieve a good balance between efficiency and performance. We can also see in Fig. 2, our method is with balanced performances in the context of all the 7 metrics.

Quantitative Evaluations

Furthermore, visual results are listed in Fig. 3 and Fig. 4. In comparison with other models, the visual results show that our model reduces the noticeable artifacts, i.e., blurring, oversharping, false contour, and incorrect color. We believe that this is attributed to the fact that our model introduces the MFI module to fully utilize spatial-temporal context information of adjacent frames to reduce these artifacts and synthesize visually pleasing textures. Besides, our approach adopts an adaptive pixel-wise kernel prediction strategy, which predicts an adaptive kernel for each pixel of the input SDR frame rather than directly estimate the HDR pixel values. More visual results can be found in our online website 1 .

4.3 Ablation Study

Effectiveness of MFI module for image-based model. In order to assess the effectiveness of the MFI module, we introduce the MFI module into the existing image-based deep learning methods (DeepHDR, KPNiTM, and HDRUnet) to capture the spatial-temporal information between adjacent SDR frames, named MFI-DeepHDR, MFI-KPNiTM, and MFI-HDRUnet, respectively. We simply introduce the MFI module into the last layer of the down-sampling for these three models. As shown in Table 2, the performance of the above three methods with the MFI module has improved. It clearly demonstrates the effectiveness of the MFI module.

1 http://www.vista.ac.cn/kpn-mfi/

Effectiveness of multi-frame interaction module. As shown in Table 3, in comparison with baseline model #1, the model #2 with the MFI module achieves and approximately yields 6.82%, 1.18%, 19.13%, and, 10.22% performance gain in terms of PSNR, SSIM, HDR-VDP3, and MABD on the testing dataset. It is clearly demonstrated that the MFI module plays a crucial role in improving performance. As the features of adjacent frames interact in the MFI module, the spatial-temporal information is captured simultaneously and the multi-frame information is leveraged to generate a frame

HDR output.

Effectiveness of brightness consistency loss. Compared with model #2, in Table 3, the model #3 not only has a performance gain of 3.25% in MABD score, but also has increases of 1.13%, 0.45%, 8.37%, and 1.98% in PSNR, SSIM, ∆EITP and HDR-VDP3, respectively. It is observed that performance can be further augmented by introducing a consistency loss. It may be due to the brightness consistency loss tends to encourage the MFI module to primarily focus on the related information between two frames and ignore the irrelevant information. Therefore, the brightness consistency loss based on Gaussian pyramid can not only improve the consistency of the model, but also help to improve the performance of the model to a certain extent.

5 Conclusion

In this paper, we propose a video iTM model which aims at reconstructing the brightness consistency of HDR video from SDR video. To this end, an MFI module is designed to capture the temporal-spatial feature from adjacent frames. Notably, we can readily extend the existing successful imagebased iTM method to capture the temporal and spatial features of the video by using the MFI module. Meanwhile, a brightness consistency loss function is adopted to further improve the inter-frame brightness consistency of the model. Extensive experimental results demonstrate that our method consistently achieves superior performance in terms of various metrics and renders visually pleasing results. We believe the proposed KPN-MFI model can be generalized to other low-level vision video tasks, which will be explored in future work.

Acknowledgements

This work was supported by the Guangdong Basic and Applied Basic Research Foundation (No. 2021A1515011584).