Beyond Feature Mapping GAP: Integrating Real HDRTV Priors for Superior SDRTV-to-HDRTV Conversion

1 Introduction

The dynamic range of a video, defined by the difference between its maximum and minimum luminance, enables High Dynamic Range (HDRTV) to deliver superior visuals. Advances in Electro-Optical Transfer Functions (EOTF), such as PQ/HLG, and Wide Color Gamut (WCG) RGB primaries (as per BT.2020), enhance HDR technology's potential.

Despite the rise of WCG-HDR displays, the production complexities result in limited WCG-HDR content. Consequently, many films remain in Standard Dynamic Range

∗Corresponding author

(SDRTV), driving the demand for SDRTV-to-HDRTV conversions. HDRTV offers a wider color gamut (Rec. 2020 vs Rec. 709), higher brightness range (0.01-1000 nits vs 0.1100 nits), advanced EOTF curves (PQ/HLG vs Gamma), and greater color depth (10-bit vs 8-bit). However, the scarcity of HDRTV content compared to SDRTV makes SDRTV-toHDRTV conversion essential, despite the inherent challenges due to the limitations of existing imaging systems and transmission protocols.

Traditional methods [Huo et al., 2013; Kovaleski and de Oliveira Neto, 2014; Ma et al., 2023] suffer from color inaccuracies and abnormal brightness restoration due to limitations in estimating curve parameters for SDRTV to HDRTV conversion. Meanwhile, recent neural network-based approaches [Kim et al., 2019; Kim et al., 2020; Chen et al., 2021b; Cao et al., 2022; Xu et al., 2022; Shao et al., 2022] employ the strategy of encoding SDRTV content into a latent space and subsequently reconstructing it as HDRTV content. Models designed by these previous methods are trained and tested on a single data set, as shown in Figure 1 (a). However, SDRTV to HDRTV conversion models trained on a single dataset are difficult to adapt to the content diversity of the real world. This challenge arises because these models learn a fixed mapping that is inherently tied to the specific characteristics of the dataset on which they are trained. Actually, these characteristics can include, but are not limited to, lighting conditions, types of scenes, and tone mapping schemes.

Insights. In the process of video production, a single HDRTV content might correlate with various SDRTV-style versions. This scenario underscores the complexity faced when training neural networks to understand not just a single linear relationship. Due to the ill-posed of multiple mapping relationships and the lack of deterministic mapping rules, it is difficult for neural networks to learn such chaotic mapping functions. This complexity highlights the substantial challenge in developing neural networks that can directly learn the diverse SDRTV-to-HDRTV conversions reflective of realworld scenarios. Therefore, it is very difficult to build a neural network to directly learn real-world SDRTV-to-HDRTV conversion.

Previous Solution. HDRTVDM[2023] emphasizes the importance of dataset diversity for training neural networks that more closely match real-world content. Although the complexity of these datasets is close to actual conditions, existing neural networks face difficulties in fully capturing and learning the diverse mapping relationships present within the dataset. This situation calls for improvements in neural network architectures to better process and understand the rich, complex data reflective of real-world scenarios.

Our Solution. In contrast, our proposed RealHDRTVNet framework enhances the quality of SDRTV to HDRTV conversion by directly embedding HDRTV priors into the transformation process, as illustrated in Fig.1 (b). This approach effectively transforms the ill-posed restoration problem into a prior selection problem, significantly reducing the solution space size. By leveraging rich and diverse HDRTV priors, our method overcomes previous limitations, achieving more accurate, generalized, and reliable SDRTV to HDRTV mapping.

Moreover, a significant challenge in SDRTV-to-HDRTV conversion is the assessment of the perceptual quality of neural network-generated HDRTV content. Common metrics like LPIPS [2018], NIQE [2012], and FID [2017], which are typically used for SDRTV quality evaluation, struggle to capture HDRTV's unique features within the PQ EOTF curve and Rec.2020 color gamut.

Inspired by this, our work extend tailored metrics for precise HDRTV quality assessment, including Learned Perceptual HDRTV Patch Similarity (LPHPS), Natural HDRTV Quality Evaluator (NHQE), and Fréchet Initial Distance (FHAD). These metrics are specifically designed to evaluate the subjective quality of HDRTV content directly. With these innovative metrics, both researchers and practitioners have the tools to conduct reliable subjective quality evaluations of HDRTV content.

This paper's contributions are in follow:

• We propose an a priori selected SDRTV to HDRTV conversion method, which significantly limits the solution space of the original high-dimensional ill-posed prob-

lem, thereby enabling efficient learning of real-world SDRTV to HDRTV conversion and improving the quality of the converted HDRTV.

• We quantitatively and qualitatively demonstrate that our proposed method outperforms previous methods.

2 RelatedWork

2.1 SDRTV-to-HDRTVMethods

Our Research Scope

Recent advancements in HDR imaging have seen the advent of various learning-based methods, as noted by Wang and Yoon [2022]. These methods have found different applications in the real world. HDR enhancement primarily involves the use of neural networks to convert SDR images to HDR [2014]. On the other hand, Multi-Exposure HDR Imaging employs exposure bracketing to create HDR images from a sequence of SDR images taken at different exposure levels [Chaudhari et al., 2019; Le et al., 2022; Xu et al., 2021; Chen et al., 2021a]. This paper concentrates on the transformation of SDRTV to HDRTV, aiming to achieve an enhanced dynamic range and wide color gamut in videos.

DNN-based SDRTV-to-HDRTV Methods

Recent studies primarily focus on devising feature modulation strategies for robust SDRTV-to-HDRTV conversion [Kim et al., 2019; Kim et al., 2020; Zeng et al., 2020; Chen et al., 2021b; Cao et al., 2022; Xu et al., 2022; Shao et al., 2022]. These strategies entail the utilization of diverse color-prior techniques to facilitate effective feature modulation. The HDRTVDM approach [2023] notably enhances HDRTV conversion quality by refining the dataset. In terms of integrated frameworks, SR-ITM [2019] presents a network design that concurrently addresses SDRTV-to-HDRTV transformation and super-resolution. Similarly, DIDnet [2023] introduces a network model that combines SDRTV-to-HDRTV conversion with the restoration of coding artifacts.

SDRTV-to-HDRTV Datasets

Within the current research landscape, a mere quartet of opensource HDRTV datasets[2019; 2021b; 2023] exists. Each dataset adheres to the BT.2020 RGB color gamut. Further, they conform to the PQ EOTF specifications with a luminance zenith of 1000 nits.

2.2 Generative Adversarial Network

The introduction of Generative Adversarial Networks (GANs), as demonstrated in Goodfellow et al.'s 2014 work [2014], has revolutionized tasks like image restoration [2021]. GAN priors, known for capturing complex image distributions, contrast traditional methods by integrating natural image characteristics into degraded visuals through adversarial training. This approach has advanced subfields like denoising, inpainting, super-resolution [2018], and artifact reduction [2020]. The strength of GAN priors lies in their restoration ability and in producing outputs closely aligned with original images. Additionally, models like VQGAN have been extensively applied in areas like multi-modal conversion [2021].

3 Methodology

3.1 Preliminary

The conversion from SDRTV to HDRTV can be mathematically represented by the Maximum A Posteriori (MAP) estimate p(h | s), which relies on the distribution of the SDRTV input p(s). Traditional methods typically utilize inverse tone mapping functions to achieve this conversion. This process is formally defined in Equation 1:

where ˆh represents the reconstructed HDRTV, θc denotes the estimated curve parameters, and ϵ accounts for the error due to parameter estimation inaccuracies. This method often leads to color distortions and anomalies in brightness restoration in the converted HDRTV content.

Modern neural network-based methods [2019; 2020; 2020; 2021b; 2022; 2022; 2022] utilize a three-stage process to complete SDRTV-to-HDRTV conversion. Initially, an encoder module is employed to map the SDRTV content into a latent representation. Subsequently, the SDRTV latent features are input to the feature transformation module to obtain HDRTV latent features. The final stage involves a decoder module, which reconstructs the latent representation back into HDRTV. By minimizing the difference with real samples, the distribution of reconstructed HDRTV is close to the distribution of real data. This dependency can be modeled as a conditional probability distribution P (h|s), which is only influenced by the distribution of the input SDRTV P(s):

where:

• θe, θτ , and θd represent the parameters of the learned encoder module, feature transformation module, and decoder module of the neural network.
• zs is the initial latent representation into which the SDRTV image s is encoded.

• ˆ zh is the intermediate latent representation obtained after encoding and feature transformation of the SDRTV image s.

These methods aim to learn a straightforward and static mapping relationship from SDRTV to HDRTV, inadequately capturing the intricate and multifaceted conversion process between SDRTV and HDRTV. HDRTVDM[2023] believes that increasing the complexity of the dataset can force the neural network to learn an SDRTV-to-HDRTV mapping function that is more in line with the real world. Although increasing the dataset is closer to the real world at the sample level, it is difficult for the neural network to directly learn such a complex mapping process.

Transitioning from SDRTV to HDRTV inherently involves conditional probability transition. On a singular dataset, these conditional transitions adhere to a stationary distribution, making it feasible for neural networks to learn such probabilistic mappings with relative ease. However, in the realworld scenario, the SDRTV-to-HDRTV conditional probability transitions do not conform to a single, fixed distribution, posing a challenge for neural architectures to learn this intricate probability transition.

To address this challenge, we propose a framework that can directly harness the prior knowledge encapsulated within HDRTV data to facilitate a superior-quality transformation from SDRTV to HDRTV. The conversion can be described by the following equation.

where:

• θ is the set of network parameters.
• zm is the best matching HDRTV prior representation selected from the HDRTV prior distribution π.
• ˆ zh is the intermediate latent representation obtained after encoding and mapping the SDRTV image s.
• π is the HDRTV prior set .
• zs is the initial latent representation into which the SDRTV image s is encoded.

To circumvent the traditional reliance solely on SDRTV inputs, our approach ingeniously integrates HDRTV priors into the conversion workflow, thereby facilitating efficient and complex HDRTV reconstruction. Our method enables adaptive conditional probability transitions based on the embedded HDRTV priors, moving beyond fixed-type conditional transitions to achieve high-quality conversion from SDRTV inputs.

3.2 Overall Framework

Following the instantiation of our motivation, we propose RealHDRTVNet, a novel architecture designed to learn the complex and variable transformations from SDRTV to HDRTV, demonstrating superior generalization capabilities in authentic scenarios. Our methodology unfolds in three phases: initially, HDRTV-VQGAN is trained to embed real HDRTV

priors. The subsequent two phase focuses on the SDRTVto-HDRTV transformation, leveraging the pre-embedded HDRTV priors to augment the quality of HDRTV. The embedded HDRTV prior can serve as a powerful guide to ensure high-quality HDRTV restoration. The proposed method no longer learns a simple function mapping that only relies on SDRTV, but learns a transformation process guided by a real HDRTV prior. Guided by real HDRTV, our method can learn complex and diverse SDRTV-to-HDRTV conversion relationships. Based on this, our method adopts a three-phase strategy:

• Phase I: We train a VQGAN model on HDRTV domain, embedding real HDRTV priors.
• Phase II: We craft a preliminary modulation encoder for SDRTV to HDRTV transformation. Through feature modulation techniques, this model refines the SDRTV latent distribution towards the anticipated HDRTV latent space, ensuring that the next stage can more accurately match the HDRTV prior.
• Stage III: We propose the HDR Color Alignment module HCA and the SDR Texture Alignment module STA. HCA identifies the best HDRTV prior from the pre-trained VQGAN codebook and uses the identified HDRTV to assist the conversion process. The SDR Texture Alignment module STA aligns the transformed features with SDRTV in texture to ensure texture fidelity.

With this three-stage approach, we provide a novel solution for high-quality SDRTV-to-HDRTV conversion.

3.3 Phase I: HDRTVVector Quantized AutoEncoder - HDR Prior Representation Learning

To reduce the uncertainty of SDRTV-to-HDRTV mapping and complement high-quality HDRTV color information, we first train a vector quantized autoencoder to learn a contextrich codebook that improves the quality of the converted HDRTV.

The specific structure is shown in Fig. 2 (a). First, the HDRTV SDRTV xh ∈ RH×W×3 is input into the encoder Eh to get the latent feature zh ∈ Rh×w×c . Next, find the closest codebook feature ˆ zh to zh in codebook C ∈ Rn×k (n is the number of vectors in the codebook) and the corresponding codebook index S. Then ˆ zh is fed into the decoder Dh to obtain the reconstructed HDRTV y′h .

We describe the details of HDRTV-VQGAN. First use the encoder Eh to encode the input HDRTV xh into a latent representation zh.

where xh ∈ RH×W×3 represents the input HDRTV, and zh ∈ Rh×w×c denotes the latent feature obtained from the encoder Eh.

Next, the replaced features ˆ zh and corresponding index sh are obtained from the codebook C through nearest neighbor matching;

where C ∈ Rn×d is the codebook containing n vectors, each of dimension d, and ˆ zh is the codebook feature closest to zh. The corresponding codebook index is obtained as S = index of ˆ zh.

Then,ˆ zh is processed via the decoder Dh get converted HDRTV:

3.4 Phase II: SDRTVModulation Encoder - Preliminary HDR Mapping

Given the substantial differences in dynamic range and color space between SDRTV and HDRTV, direct alignment of SDRTV features to HDRTV priors in latent space is inherently challenging. To mitigate this, we decompose the prior matching problem into a two-stage process.

First, we propose the SDR Feature Modulation Encoder (Esfm) to transform SDRTV latent features into a space that is more congruent with HDRTV priors. The encoder Esfm is composed of four sequential SDRTV Feature Modulation (SFM) blocks, which iteratively refine the latent representation zsfm to approximate the HDRTV distribution.

Formally, the input SDRTV frame xs (referred to as x0 here) is processed as follows:

where i ∈ {1, 2, 3, 4}, and c ∈ {64, 128, 256, 512}, representing the number of channels at each respective stage. In detail, the SFM is implemented by:

where ⊙ denotes element-wise multiplication, αi and βi denote the modulation parameters derived via convolutional layers Convθ.

Subsequently, a Transformer module aggregates the modulated features into a compact latent representation:

This process effectively aligns the SDRTV latent space with that of HDRTV, thereby facilitating optimal prior matching in subsequent processing stages.

3.5 Phase III: RealHDRTVNet - High-Quality Conversions with Pretrain HDR Prior

In phase I and II, Esfm, the codebook C, and Dh are pretrained and then frozen in the subsequent phase. In phase III, to achieve efficient and nuanced HDRTV reconstruction, we propose RealHDRTVNet, which consists of two key components: HDR Color Alignment Module(HCA) and SDR Texture Alignment(STA). These components ensure that the converted HDRTV retains the texture structure of the input SDRTV while aligning with the dynamic range and color of real HDRTV.

HDR Color Alignment Module(HCA)

To achieve accurate color alignment with real HDRTV priors, we introduce the HDR Color Alignment module(HCA), which integrates the functionalities of brightness prior extraction, prior matching, and prior adaptive instance normalization.

Given an SDRTV input xs, the encoder Esfm first extracts the multi-scale feature zmsf and the "basic" HDRTV feature zsfm. Recognizing the necessity for adaptive processing in highlight regions, we employ a brightness prior extraction module to generate a luminance-aware position coding zc using Xs and its highlight mask. This feature zc is then fed into the Prior Matching (PM) module, where it is aligned with the optimal HDRTV prior ˆhhm from the codebook C, resulting in the matched feature ˆ zhm.

After obtaining ˆ zhm, Prior Adaptive Instance Normalization (AdaIN) adjusts the latent features of zsfm and ˆ zhm, generating ˆ zhma. The feature ˆ zhma is then fed into the pretrained Dh to produce the HDRTV result. This process ensures that the color and dynamic range of the generated HDRTV align with real HDRTV priors.

SDR Texture Alignment Module(STA)

. To maintain the texture structure of the input SDRTV in the converted HDRTV, we introduce the SDR Texture Alignment module. This module uses multi-scale features zmsf from the encoder Esfm during the decoding process to align textures.

First, the multi-scale features zmsf from the encoder are concatenated with the decoder features ˆ zd and fed into a deformable convolution to achieve alignment, resulting in aligned features ˆ zn. These aligned features are then used to modulate the decoder features ˆ zd, resulting in the aligned decoder feature ˆ zda, which is fed into decoder Dh for HDRTV reconstruction.

By performing SDR Texture Alignment on the decoder at four different resolutions, the decoder Dh is capable of recovering HDRTV with a realistic dynamic range while preserving high-quality texture details from the SDRTV.

Overall Pipeline

The complete RealHDRTVNet pipeline is described as follows:

4 Experiment

4.1 EvaluationMetrics

We use objective metrics, subjective metrics and user study to evaluate different methods. Objective metrics include PSNR, SSIM and HDRVDP3, which respectively evaluate the fidelity, structural similarity, color fidelity, and visual similarity of the converted HDRTV. Subjective metrics include LPHPS(Extended from LPIPS), FHAD(Extended from FID), and NHQE(Extended from NIQE), which can evaluate the perceptual similarity and distribution consistency of the converted HDRTV and the real HDRTV. These three subjective metrics are obtained by expanding the previous subjective quality assessment methods in this paper.

4.2 Implementation Details

We validate our method's efficacy by training and testing on various datasets detailed in Section 2.1, and additionally evaluating real SDRTV datasets. Our approach employs the Adam optimizer with an initial learning rate of 2 × 10-5 , utilizing a cosine annealing schedule for learning rate decay. Training is divided into three phases, which means HDRTVVQGAN, Esfm, and RealHDRTVNet are trained respectively.

4.3 Comparisons with State-of-the-Art Methods

We compared our proposed RealHDRTVNet with state-of-the-art methods, including HDRTVNet[2021b], HyCondITM[2022], HDCFM[2022], HDRTVDM[2023], FMNet[2022], and DEEPSRITM[2019]. Extensive experiments were conducted on both synthetic and real-world datasets.

Experimental Results on Synthetic Datasets

We first show quantitative results on three synthetic datasets in Table 1. On quality metrics PSNR, SSIM, LPHPS, FHAD, and NHQE, our RealHDRTVNet achieves the best scores than existing methods. This means that our method achieves the best performance and produces HDRTV results of higher subjective quality.

Our method achieves a lower value on the LPHPS metric, indicating that the perceptual difference between the proposed method and the real HDRTV is smaller, making it closer to the actual HDRTV. Additionally, the reduced FHAD and NHQE metrics suggest that the HDRTV produced by our method better matches the distribution of HDRTV captured from real-world scenes.

Furthermore, we performed a qualitative comparison in Fig.3. Previous methods failed to produce satisfactory conversion results, such as HDRTVDM and FMNet. Our method can convert HDRTV with more realistic and natural highlights and is particularly effective in reinstating natural and continuous color gradients in areas experiencing color transitions. This capability ensures that the images not only exhibit greater visual fidelity but also reflect a smoother and more authentic representation of real-world colors and shading, enhancing the overall viewing experience.

Visual Perceptual Quality Assessment via HDRVDP3 metrics

Table 3 presents a comparative performance analysis of various methods on the HDRTV1K, HDRTV4K, and SRITM datasets using the HDRVDP3 metric.

Our method achieves the highest HDRVDP3 scores on the HDRTV1K (8.28), HDRTV4K (7.91) and SRITM(7.8) datasets, indicating superior perceived quality of the HDRTV content.

Experimental Results on Real Datasets

To verify the generalization of our method in the real world, we evaluate it on multiple real-world datasets, including BSD100 [2001], CBSD68-Noisy [2001], CBSD68-Original [2001], Set5 [2012], and Set14 [2012]. As shown in Table 2, our RealHDRTVNet achieves the best FHAD score as well as NHQE perceptual quality on real-world SDR datasets.

4.4 Ablation Study

To assess the impact of the proposed components, we start with a baseline and systematically integrate each component one by one. In Table 4, we can observe that each component has brought improvement, and the improvement in the PSNR metric reached 0.14. Meanwhile, our visual ablation results, presented in Fig. 4, demonstrate that the modules we designed significantly enhance visual quality.

Importance of SDRTV Feature Modulation Encoder

Esfm

We first study the effectiveness of feature modulation encoders. As shown in the third line of Table 4, deleting the modulation module in the encoder will cause the LPIPS and PSNR metric to deteriorate.

Effectiveness of HDR Color Alignment HCA and SDR Texture Alignment STA

We ablate the brightness prior extraction module HCA and the SDR Texture Alignment module STA. As can be shown in Table 4, after adding HCA and module STA in sequence, the performance of our model improved.

5 Conclusion

In this work, we introduce a novel paradigm for SDRTVto-HDRTV conversion: HDRTV prior-guided high-quality SDRTV-to-HDRTV transformation. In contrast to traditional approaches that solely rely on SDRTV, our method achieves a more realistic and superior quality in HDRTV reconstruction. Additionally, we extend commonly used subjective quality evaluation metrics in SDRTV, such as FHAD, NHQE, and LPHPS, to assess the quality of HDRTV. Our proposed technique exhibits significant improvements in visual quality.

Acknowledgements

This paper is supported by the construction project of the Fujiang Laboratory Nuclear Medicine Artificial Intelligence Research Center (No. 2023ZYDF074) , supported by the Fundamental Research Funds for the Central Universities under Grant ZYTS25270. Gang He and Kepeng Xu contributed equally to this work. Please ask Dr. Kepeng Xu(kepengxu11@gmail.com) for correspondence.