MetaJND: A Meta-Learning Approach for Just Noticeable Difference Estimation

1 Introduction

In visual signal analysis, just noticeable difference (JND) measures the smallest change or difference that the human visual system (HVS) can perceive. Such measurements effectively quantify the visual redundancy existing in visual signals [Bai et al., 2022], which can be used to achieve perceptual coding [Cheng et al., 2021], thereby saving more bit-rate without damaging visual quality [Nami et al., 2022; Wang et al., 2021]. This is of great importance due to the increasing demand for high visual quality in digital visual services like online video streaming, social media sharing, virtual reality, etc. Furthermore, JND can also be widely used in various perceptual image and video processing tasks, such as enhancement [Nikzad et al., 2021], compression [Qi et al.,

∗Corresponding author: Wuyuan Xie

2023], digital watermarks [Qu et al., 2023], and quality assessment [He et al., 2022].

Traditional JND models are typically driven by the HVS knowledge. Knowledge-driven approaches primarily compute JND by combining various HVS visual effects, such as brightness adaptation [Yang et al., 2005], edge and texture sensitivity [Liu et al., 2010], free energy principle [Wu et al., 2013], pattern complexity diversity [Wu et al., 2017], central foveal effect [Chen and Wu, 2019], feedforward and feedback modulation effects [Yin et al., 2023], and blur sensitivity effect [Wang et al., 2022]. However, how to accurately simulate multiple visual effects and their interactions remain challenging for these methods due to the limited knowledge of the HVS.

Recently, data-driven JND models [Shen et al., 2020] have emerged as a new trend due to their ability to automatically learn descriptive features from visual data. The performance of these approaches heavily relies on the amount and distribution of datasets. However, there is a limitation in popular JND datasets such as PWJND [Shen et al., 2020] and KonJND1k [Lin et al., 2022], as they have very limited data available due to the high cost of subjective JND labeling. This limitation can greatly impact the accuracy and generalization of the training models. To address this issue, [Xie et al., 2023] proposed a multi-modal network called hmJND-Net for the JND estimation, which utilized RGB, depth, saliency, and segmentation modalities to solve the problem of data scarcity.

Motivated by the discussions above, this paper presents a multi-modal-driven meta-learning JND framework, namely MetaJND, to improve the accuracy and generalization. There are two key differences between MetaJND and [Xie et al., 2023]. First, [Xie et al., 2023] includes the segmentation modality to provide object category information. However, this can lead to issues when attempting to match category information with the JND threshold, potentially misleading model training. For example, as depicted in Figure 1, different buildings are with the same object category, but they have different JND thresholds. In contrast, MetaJND avoids this problem by excluding the segmentation modality. Second, [Xie et al., 2023] combines auxiliary modalities and aligns them with the RGB modality. This approach faces two potential issues: 1) Only a few top-layer features are fused and aligned, while a significant number of features from different layers are ignored. 2) It is difficult to guarantee all auxiliary modalities are efficiently aligned and utilized. In contrast, MetaJND first employs the attention mechanism and metalearning to separately align the features of the RGB and each auxiliary modality. Then, we introduce a multi-layer channel and spatial feature rectification to fuse the aligned features, which can comprehensively utilize the information of hierarchical features.

The main contributions of this paper are threefold: 1) To improve the limited generalization of deep JND models, we have proposed a meta-learning JND model at the pixel-level, namely MetaJND. By using meta-learning, we can make online adjustments to the RGB, saliency, and depth modalities, ensuring a higher generalization. 2) To effectively align features from different modalities, we have devised a novel multi-modal meta alignment (MMA) module that combines attention mechanism and meta-learning. This module combines attention mechanisms and meta-learning techniques to efficiently align the RGB features with each auxiliary modality. 3) To fully exploit complementary information from hierarchical features, we have designed a multi-layer channel and spatial feature rectification (MCSFR) module and a multiscale feature fusion (MSFF) module to merge the aligned multi-modal features, predicting more accurate JND.

2 RelatedWork

In this section, we first briefly review some representative data-driven JND models, including single-image and multiimage modality JND methods. We then provide a detailed explanation of the motivation behind our MetaJND.

2.1 Single-Image ModalityMethods

The single-image modality JND models are typically trained on the original RGB data and their distorted versions. This type of method predicts JND in either picture or block level with deep networks, like convolutional neural networks (CNN). Picture-level JND (PJND) prediction distinguishes the critical lossless image from several distorted images with different levels of noises. For instance, perceptually lossless/lossy binary classifier [Liu et al., 2019], satisfied user ratio (SUR) prediction [Lin and Ghinea, 2022], and quality factors prediction [Tian et al., 2020] have been developed for the PJND estimation. In contrast, block-level JND (BJND) prediction is conducted on the image blocks instead of the whole picture. For instance, block-level structural degradation [Shen et al., 2020] and four-stage block-level framework [Nami et al., 2022] have been introduced to predict BJND.

2.2 Multiple-Image ModalityMethods

The multiple-image modality JND models aim at utilizing not only the RGB image information but also auxiliary modalities to predict the JND. For example, [Wu et al., 2020] explored the JND modeling by introducing visual attention and pattern complexity. Based on the gradient modality and the class activation mapping (CAM) modality, [Jin et al., 2021] designed signal degradation networks to simulate human perception. Additionally, [Xie et al., 2023] fused aligned depth, saliency, and segmentation modalities to estimate the JND threshold.

2.3 Motivation

Due to the lack of sufficient data in existing JND datasets, both single-image modality and multiple-image modality JND models have been developed to improve the prediction performance. The former simplifies the JND estimation by predicting at the picture or block level, while the latter increases the amount of data by incorporating auxiliary modalities. Among them, multiple-image modality methods show more promise in accurately estimating pixel-level JND.

However, the modeling of the JND framework using multiple-image modality is still in its infancy. The main challenges lie in the alignment and fusion of multiple modalities that may differ significantly from each other. Traditional learning strategies struggle to simultaneously align different pairs of modalities, such as RGB-depth and RGBsaliency, due to their distinct characteristics. To address these challenges, we employ meta-learning [Hu et al., 2019; Zhang et al., 2021; Ma et al., 2022; Ye et al., 2021] to model the JND framework, which enables adaptability to the features of various modalities. We believe that employing meta-learning cannot only improve the alignment of different modalities, but also enhance generalization capabilities.

3 ProposedMethod

In this section, we present the details of the proposed MetaJND. As shown in Figure 2, our MetaJND mainly consists of a multi-modal alignment (MMA) module, a multi-layer channel and spatial feature rectification (MCSFR) module, and a multi-scale feature fusion (MSFF) module.

3.1 ProblemFormulation

The multi-modal data typically originate from different sensors or data sources, providing distinct visual information and features. However, these modalities also have distinct representations, distributions, and feature spaces. For instance, the RGB modality covers complete but low-level information, while the depth and saliency modalities contain incomplete but high-level semantic information. Therefore, it is necessary to align the extracted shallow features from different modalities using an alignment network Falign. This alignment is achieved by applying the same loss function, Lalign, to both the RGB modality and each auxiliary modality. The aligned features are then fed into a fusion network, Ffuse. The JND threshold is predicted by upsampling the fused features. The optimization problem is to minimize the mixed loss of Lalign and the final pixel distance Lpix between the prediction and ground truth:

3.2 Multi-modal Meta Alignment

To align multi-modal features for effective information fusion, we propose a multi-modal meta alignment (MMA) module, which consists of a dual-modality alignment (DMA) module and a meta-learning module (MM), as illustrated in Figure 3. DMA aligns the shallow feature of the RGB and the auxiliary saliency or depth modality, while MM further makes adaptive adjustments to the aligned features, which improve the generalization of MetaJND.

Dual-Modality Alignment

In the dual-modality alignment (DMA) module, we employ the Swin Transformer (ST) to align each pair of modalities. Considering both its complexity and efficiency, we have Dual Modality Alignment (DMA)

Meta-learning Module (MM)

Figure 3: Illustration of the proposed multi-modal meta alignment (MMA). The inputs are the RGB modality feature Frgb and the auxiliary features Faux that consists of Fdep and Fsal, while the outputs are the meta-aligned features ˆFidep and ˆFisal, respectively.

adopted the Swin-B version [Liu et al., 2021]. Before the DMA, we apply a 3×3 convolution network to individually extract shallow features (i.e., Frgb, Fdep, and Fsal) from the RGB, depth, and saliency modalities (i.e., Irgb, Idep, and Isal). The shallow features are then fed into the DMA module, which adopts a dual-stream structure to align the RGB and auxiliary modality features. Additionally, the multi-head cross-attention (MCA) is incorporated to guide the network to consider the correlation of different modalities and learn their interrelations. In practice, the RGB and auxiliarymodality features are separately passed through two consecutive ST blocks with a windowed multi-head attention (W-MCA) and a sliding window multi-head attention (SW-MCA). Q, K, and V are computed by mapping through the weight matrix W, and the calculation process is as follows:

where X represents Fdep or Fsal, and Conv(·) denotes the convolution operation.

Subsequently, we add a multi-layer perceptron (MLP) and a GELU layer to enhance the feature tokens generated by the MCA. A layer normalization (LN) with the residual connections is added before the MCA and MLP blocks. Consequently, the entire process can be expressed by

where Fimca and Fistb represent the output features of the MCA blocks and the i-th ST block, respectively.

Meta-learning Module

When performing the DMA, shallow features from three modalities are mapped to a shared feature space. However, the limited JND-specific dataset hinders training a network to create a complete feature space that is suitable for unseen inputs, resulting in low generalization ability. To help the network learn a more general representation, we introduce a meta-learning module, which can generate adjusters for different inputs.

(a) Unaligned features

MM is mainly comprised of a channel attention block and a meta-net. The channel attention block explores dependencies between the two channels to generate attention features, denoted as Fatt = Fatt(Fistb ). For convenience, we take SEnet [Hu et al., 2018] as the backbone of Fatt. The metanet is constructed with two fully-connected (FC) layers, with a ReLU layer in between. The adjusters generated by the meta-net are based on Fatt. During the feed forward process, the first FC layer extracts features from the inputs, while the second FC layer reshapes these extracted features into an adjuster, Fmeta. The process of generating adjusters can be defined by

where FC(·) and ReLU(·) represent the FC and ReLU layer, respectively. Φ(·) represents the reshape operation.

Fmeta enables the network to adjust to different attention features dynamically. Specifically, it uses the adjuster to assign weights to the aligned features in the DMA, followed by a convolution operation. After the MM module, the output features of the MMA module are represented by

where ˆFdep/sal represents the final aligned RGB-depth or RGB-saliency feature, and Concat(·) represents a concatenation operation.

To verify the effectiveness of the proposed MMA module, we have conducted a t-SNE visualization of aligned and unaligned features as depicted in Figure 4. In the scatter plot before alignment, the features of the three modalities form separate clusters, with each cluster representing a different modality. This indicates that the distributions of the different modalities are relatively independent when not aligned. However, after performing the MMA, the features of the three modalities become clustered together and overlap. This finding suggests that the proposed MMA has the ability to map the initially dispersed modality features into a shared feature space, resulting in a more concentrated feature distribution.

3.3 Multi-layer Channel and Spatial Feature Rectification

The fusion process, Ffuse, plays a crucial role in fully utilizing the multi-modal information to improve the estimation performance. However, existing methods tend to only fuse features from the top layer but discard hierarchical features from the intermediate layers. As the high-layer has larger receptive fields, it generates coarse-grained features that capture the overall shape and structure. In contrast, the low-layer benefits from smaller receptive fields and can extract fine-grained features with more details. Additionally, the aligned features ˆFdep and ˆFsal may still contain noises due to the uncorrelated features extracted from different modalities.

To address these issues, we propose a multi-layer channel and spatial feature rectification (MCSFR) module. This module extracts multi-scale features and rectifies them before performing Ffuse. It contains a feature transformation (FT) module and a feature rectification (FR) module. The FT module utilizes ST blocks and patch merging layers to transform features into different scales. The FR module corrects latent features with a multi-layer channel rectification module and a spatial feature rectification module. Our MCSFR is shown in Figure 5.

Multi-layer Channel Rectification

Multi-layer channel rectification (MCR) aims to filter the noise in different channels. Specifically, we both apply the global max-pooling and the average-pooling to both the aligned saliency feature ˆFjsal and depth feature ˆFjdep , where j means the j-th transformation. This module retains more information and better represents different aspects of the input modalities. Subsequently, the latent features are concatenated into a single tensor ˆFjmcr . Then, ˆFjmcr is fed into a multilayer perceptron (MLP) and a Sigmoid layer, and two sets of trainable weights (i.e., Wc,j dep andWc,j sal ) are obtained for rectifying the original features. This process can be expressed as follows:

where Fsplit(·) refers to the process of splitting the output of MLP into two parts. Then, the multi-layer channel rectification process is formulated as:

where ⊗ denotes an element-wise multiplication, Dc,j rec represents the rectified RGB-depth features, and Sc,j rec represents the rectified RGB-saliency features after the multi-layer channel rectification.

Spatial Feature Rectification

In the process of the spatial feature rectification, the input features are first concatenated together. Then, the concatenated spatial feature, Fjsfr , is obtained along the channel dimension through an MLP, with the calculation formula as follows:

Similarly, Fjsfr is fed into a Sigmoid layer and divided into two parts, Ws,j dep andWs,j sal . The detailed process is expressed as:

The spatial rectification is then expressed by

where Ds,j rec and Ss,j rec represent the rectified RGB-depth and RGB-saliency features after the spatial rectification, respectively.

Finally, the overall rectified features ˜Fjsal and ˜Fjdep after the j-th transformation are obtained by

where λc and λs are two hyper-parameters, and they are set to 0.5. j belongs to {0, 1, 2, 3}.

3.4 Multi-Scale Feature Fusion

After the MCSFR module, the rectified features are further fused for the final JND estimation. To fully take advantage of the multi-scale information provided by different layers, we have designed a multi-scale feature fusion (MSFF) module, which consists of four dual modality fusion (DMF) modules. DMF first fuses the multi-scale rectified features ˜Fjdep and ˜Fjsal using addition, multiplication, and concatenation opera-

where Fjcat represents the early fused features after the concatenation.

Next, we progressively aggregate the early fused features and upsample them into shallow features for the final JND estimation, which can be expressed as:

where Upsampling(·) denotes a 2× upsampling, which employs a bilinear interpolation. The predicted JND map is then obtained by feeding the fused feature Ffuse into an unsampling network.

4 Experimental Validations

In this section, we present the implementation details of our MetaJND and report the experimental comparison with eight representative methods. Moreover, we conduct an ablation study on the impact of using different modalities and modules.

4.1 Experimental Protocols

Dataset Description

In the experiments, we have trained the MetaJND using the benchmark dataset PWJND [Shen et al., 2020]. This dataset contains 202 source images with a resolution of 1920×1080, where each original image is labeled with a compressed version with Versatile Video Coding (VVC). We randomly split the whole dataset into training, validation, and test sets with a ratio of 8:1:1.

To verify the generalization ability of different models, we further select three additional JND benchmark datasets, including MCL-JCI [Wang et al., 2016], KonJND-1k [Lin et al., 2022], and MDTJND [Liu et al., 2023] for testing. In addition, we employ BASNet [Qin et al., 2019] and DPTHybrid [Ranftl et al., 2021] to generate the saliency and the depth modalities, respectively.

Training Details

Our MetaJND-Net is implemented on the PyTorch with all weights initialized using a truncated normal initializer. We use the default parameters of the Adam optimizer, such as β1 = 0.9 and β2 = 0.999. Before the training, we crop the input images into 224×224 with a random cropping and a random rotation. We set the batch size to 8 and the initial learning rate to 1e-5. The model training is then conducted on NVIDIA GeForce RTX 3090 GPU, which takes approximately 16 hours for 200 epochs.

To guarantee the alignment efficiency, we have developed a meta-alignment loss Lalign, which is calculated by a pixelwise L1 distance between the input RGB feature Frgb and the aligned feature ˆFy. With α setting to 1, the total loss function in (1) can be expressed as:

where Igt refers to the ground-truth, while Ijnd denotes the predicted JND map.

Evaluation Settings

We have conducted the commonly-used noise injection experiments [Xie et al., 2023; Jiang et al., 2022] to evaluate the accuracy of the JND estimation. Specifically, we randomly injected noise into each pixel with the guidance of Ijnd, which can be formulated as:

where Inoised represents the JND-guided noised image, β randomly takes ±1, and µ is an adjuster scalar. A better JND model can tolerate more noise, thereby achieving better visual quality under the same noise level. In our experiments, we adjust µ to control the amplitude of the injected noise, aiming for a specific peak signal-to-noise ratio (PSNR). Specifically, the noise levels of the first JND level for four datasets are PSNR=35.05dB, 33.18dB, 38.35dB, and 35.12dB, respectively. To achieve consistency, we control the injected noise level to the mean value (e.g., PSNR=35.42 dB) of these four datasets.

To facilitate quantitative comparisons, we have used three additional widely-used perceptual metrics to assess the image quality: the structural similarity index (SSIM), the multiscale structural similarity (MS-SSIM), and the learned perceptual image patch similarity (LPIPS) [Zhang et al., 2018].

4.2 Performance Comparisons

To demonstrate the overall performance, we provide the qualitative and quantitative results of our MetaJND and eight representative methods, including Liu2010TCSVT [Liu et al., 2010], Wu2013TIP [Wu et al., 2013], Wu2017TIP [Wu et al., 2017], Chen2019TCSVT [Chen and Wu, 2019], Shen2020TIP [Shen et al., 2020], Jiang2022TIP [Jiang et al., 2022], Wang2022TII [Wang et al., 2022], and Xie2023AAAI [Xie et al., 2023].

Subjective Qualitative Results

Figures 6 (b)-(j) show the subjective visual comparison results for JND estimation of nine methods, where brighter pixels indicate higher tolerance redundancy. In (b)-(i), higher visual redundancy is observed in smooth background regions

(highlighted by the red box) and intricate structural regions (highlighted by the green box). As human vision exhibits higher perceptual sensitivity to uniformly smooth and structural regions, lower visual redundancy should be estimated in these areas. Consequently, our MetaJND, benefiting from meta-learning and multi-layer fusion, achieves more precise JND estimation in these sensitive regions.

Objective Quantitative Results

Table 1 presents the SSIM, LPIPS, and MS-SSIM results for nine JND models. Our MetaJND demonstrates stable performance across all four datasets, demonstrating its superior capability in noise tolerance and generalization to different types of data.

In addition, to address concerns of fairness in the use of the noise injection level at the PSNR=35.42dB, we have expanded our experiments to include a wider range of noise levels for performance comparison. For instance, we have considered the noise injection level at various PSNRs, including {32, 33, 34, 35, 36, 37, 38, and 39}. In Figure 7, we have plotted the performance curve. As seen, our MetaJND consistently outperforms the other methods.

4.3 Ablation Study

In this section, we have conducted the ablation experiments to verify the effects of two additional modalities and our designed modules on the dataset PWJND [Shen et al., 2020]. The selected (unselected) modality or module is denoted by "✓" ("×").

Effects of Auxiliary Modalities

We have conducted four experiments to validate the necessity of the depth and saliency modalities as shown in Table 2. For example, to evaluate the impact of the depth modality, we replace it with the RGB modality. As seen, selecting all three modalities simultaneously leads to the greatest performance improvement, highlighting the necessity of these two auxiliary modalities.

Effects of MMA, MCSFR, and MSFF

To demonstrate the importance of the proposed MMA, MCSFR, and MSFF modules, we have also conducted a series of ablation experiments. The results are provided in Table 3. In the experiments, we disable the unselected module and replace it with a simple network. For example, the MMA module is replaced by a network that incorporates a concatenation operation, three 3×3 convolution layers, and one 1×1 convolution layer. The MCSFR module is replaced with a three-layer ST block, while the MSFF module is simply replaced with concatenation. As seen, the best performance is achieved when all three modules are enabled simultaneously.

5 JNDApplications

5.1 Compression Application

JND has been widely-applied in image coding to enhance the perceptual quality. In this section, we have integrated MetaJND into three encoding standards, including JPEG, HEVC, and VVC for compression applications. We have employed the same compression methods as described in [Xie et al., 2023]. Figure 8 shows the visual comparisons of JND-guided compression. As seen, all JND-guided compression methods provide nearly identical visual quality. However, our MetaJND approach achieves the highest bit-per-pixel (BPP) savings, with an average of 24.11% coding gain on four datasets.

5.2 Watermarking Application

Image watermarking is commonly used to achieve copyright protection, content verification, and identity authentication.

As suggested in [Fang et al., 2023], the JND map is introduced as a weight matrix in the loss function in the luminance channel to improve the deep watermarking model. The robustness of watermarking is measured by the bit error rate (BER) performance. We evaluate the watermarking performance under four types of attacks, including JPEG (quality factor (QF) =50), Gaussian filter (kernel size (KS)=2), Gaussian noise (σ2=0.003), and Salt & Pepper (density=0.01). As seen in Figure 9, our MetaJND-guided method archives lower BER than [Fang et al., 2023] under the same PSNR level as in Table 1. The average BER value of our MetaJNDguided method is 0.0913 on four datasets, while that of [Fang et al., 2023] is 0.0951.

6 Conclusion

In this paper, we propose a new meta-learning framework, called MetaJND, for estimating just noticeable differences at the pixel level. MetaJND leverages multiple modalities, including RGB, saliency, and depth, to enhance both its accuracy and generalization. To achieve this, we have developed a multi-modal meta alignment module that combines attention mechanisms and meta-learning to align features from different modalities. Additionally, we have explored a multi-layer channel feature rectification module to correct intermediate features before conducting the multi-modal fusion. Furthermore, we have designed a multi-scale feature fusion module to merge the aligned features obtained from different layers. Experimental results on four benchmark datasets, along with comparisons to eight representative JND methods, demonstrate the superior performance and enhanced generalization capabilities of our MetaJND.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 61701310 and 62372306, in part by the Natural Science Foundation of Shenzhen City under Grant 20220809160139001, and in part by the Natural Science Foundation of Guangdong Province under Grants 2024A1515011972, 2021A1515011877, and 2022A1515011245.