HIPP: Protecting Image Privacy via High-Quality Reversible Protected Version

1 Introduction

In recent years, with the rapid development of the internet, more and more people tend to share various images about their lives on Social Network Platforms (SNPs). In 2023, there were 5.07 billion social media users worldwide [Petrosyan, 2024], with approximately 2 billion users uploading photos on Instagram monthly [Dixon, 2024] and around 243, 055 new photos being uploaded to Facebook every minute [Kevin, 2024]. Although SNPs provide convenient sharing and communication, bringing users closer together, they have introduced a range of privacy issues as images often contain users' personal private information such as identity and location [Such and Criado, 2018; Morales et al., 2021; Fan, 2019; Zeng et al., 2015; Li et al., 2024]. For one thing, malicious users may analyze the content of images on SNPs to obtain private information of users [Barnes, 2006; Narayanan and Shmatikov, 2009]. For another, SNPs themselves may be curious about the user information contained in the uploaded images [Isaak and Hanna, 2018; Mazzarolo et al., 2021; Ra et al., 2013]. As traditional image encryption ensures that all information in the original image is invisible, providing maximum privacy protection but limiting the sharing experience of non-sensitive information [Singh and Singh, 2022; Krishna et al., 2022; Deng et al., 2023]. Partial image protection makes detected sensitive areas invisible while keeping the resting unchanged for original visual effect preservation [Beugnon et al., 2019; Morris et al., 2023; Zhang et al., 2023a; He et al., 2024a]. Unfortunately, the sensitive areas in images are prone to omission, posing a risk of privacy leakage. Another popular technology for privacy protection is named face anonymization[Maxim et al., 2020; Li et al., 2023; He et al., 2024b], which works by modifying facial features to alter the appearance. However, this technology only focus on faces, ignoring that other content in shared images may reveal sensitive personal information such as current location and home address.

∗Lina Wang and Run Wang are corresponding authors

In view of this, a novel thumbnail-preserving image privacy protection technique has been proposed, which is usually categorized into two types: 1) complete thumbnail preservation [Wright et al., 2015; Tajik et al., 2019; Chai et al., 2022; Zhang et al., 2022c]; 2) approximate thumbnail preservation [Marohn et al., 2017; Zhang et al., 2022b; Ye et al., 2023; Zhang et al., 2023b; Chowdhury et al., 2024; Zhao et al., 2024]. On the one hand, the technique protects all image details within thumbnail blocks by disturbing the specific pixel values to prevent malicious users from extracting privacy information from the details of images displayed on SNPs. On the other hand, it preserves the overall visual effect of the original image for online browsing and management by ensuring that the protected image has a same or approximate thumbnail to the original one. As shown in Figure 1(a), the uploading user, i.e., the image owner, processes the original images locally and uploads the protected images into SNP. The malicious user, who is curious about the privacy information of other users, may collect a large number of images displayed on SNP but cannot analyze the detail privacy information within them. Although the existing works achieve the goal of protecting image privacy while retaining overall visual effect, the generated protected image often contains a lot of noise as shown in Figure 1(a), leading to poor visual perception and poor sharing experience. Meanwhile, malicious users can easily aware that the image is protected and target it for attacks.

Motivated by the observation that the latent vector corresponding to the image can be decoupled into the detail and contour parts, in this paper, we propose HIPP, a reversible scheme based on latent vector decoupling to achieve image privacy protection for social images sharing. Unlike the existing schemes, our scheme applies a detail information extractor to decouple the vector into the detail and contour vectors in the latent space instead of directly utilizing encryption or other obfuscation techniques in the image space. The new vector is composed of the original contour vector and the replaced detail vector, which can be utilized to generate the protected image with high visual quality by generation models. Moreover, as protected images are indistinguishable from the real images, the malicious user cannot aware that the protected images displayed in SNP are fake images and analyze the fake privacy data. The main contributions of our scheme proposed in this paper are as follows:

• We propose a novel reversible image privacy protection scheme that generates naturally appealing protected images that are indistinguishable from real natural images, improving visual quality for better sharing experience and misleading malicious users into obtaining incorrect personal private information.
• We design a detail information extractor that can decouple the detail and contour information contained in the latent vector corresponding to the image. Meanwhile, generation models are applied to reconstructing the protected images with different details, which pro-
• vides a new perspective for future research in the area of thumbnail-preserving image privacy protection.
• Experimental results demonstrate that the generated protected images appear natural without any visible noise. Moreover, our scheme achieves the lowest runtime and file expansion rate compared to existing methods, improving its practical value.

2 RelatedWork

Image Privacy Protection for SNPs

As the earliest method for image privacy protection, traditional image encryption schemes [Singh and Singh, 2022] encrypt the whole image into a snowflake-like ciphertext by the delivered key, preserving no visual information related to the original image and limiting content sharing. Consequently, a partial image privacy protection approach, which only protects areas related to privacy within the image, has gained popularity. Initially, these areas are protected by simple techniques such as blurring (adding noise) [Neustaedter and Greenberg, 2003; Neustaedter et al., 2006], pixelation (replacing pixels in the same sub block with their average value) [Lander et al., 2001; Zhou and Pun, 2021], masking (covering all pixels by a specific value) [Park and Kim, 2020; Yu et al., 2020], or object removal (filling with new content related to surrounding pixels) [Yi et al., 2020; Wang et al., 2021], which do not take multiple privacy conflicts into account. To address this, a hierarchical privacy protection scheme for privacy image sharing was proposed, allowing the reconstruction of the original version only when a sufficient number of individuals consent [Beugnon et al., 2019]. In another approach, Liu et al. designed a system to achieve automated facial protection by learning sensitive relationships among users [Liu et al., 2022]. Subsequently, the issue of privacy leakage caused by image forwarding still persisted, which was later addressed by a cross-platform protection scheme based on blockchain technology [Zhang et al., 2022a; Zhang et al., 2023a]. A crucial aspect of partial image protection mentioned above is the detection of privacy-sensitive areas within the image. However, no existing approach can guarantee that all sensitive areas can be found.

Thumbnail-Preserving Image Privacy Protection

The objective of thumbnail-preserving image privacy protection is to make details within thumbnail blocks invisible while preserving the whole image thumbnail unchanged, which was first introduced by Wright et al. (2015). This scheme divides each thumbnail block into several sub blocks, and then rearranges pixels in every sub block, followed by rearranging sub blocks in the same thumbnail block which is sample but exposes the pixel lists of thumbnail blocks. Taijik et al. 2019 designed a sum-preserving encryption (SPE) algorithm to encrypt every two pixels while keeping their sum unchanged. Therefore, the sum of each block after encryption is completely identical to the original one. Although the scheme achieves nonce-respecting security, its time cost is quite high. Hence, a scheme named TPE-GAN was proposed to improve the efficiency by applying CycleGAN [Zhu et al., 2017] to simulate randomized unary encoding [Chai et al., 2022]. At the same time, F-TPE came up with a multi-pixel SPE algorithm, achieving fast calculation on vector sets [Zhang et al., 2022c].

Compared with completely thumbnail-preserving privacy protection schemes mentioned above, approximately thumbnail-preserving schemes achieve high efficiency at the expense of similarity between the protected and original thumbnails. Morohn et al. (2017) proposed two schemes named DRPE and LSB-TPE, which have possibility of decryption failures or incomplete recovery. Therefore, HF-TPE was proposed to ensure successful decryption, with the decrypted image being closer to the original one by constructing a sum-preserving data embedding algorithm [Zhang et al., 2022b]. Subsequently, Ye et al. (2023) and Zhang et al. (2023b) proposed two completely reversible schemes to confirm that the restored image is identical to the original one by combining reversible data hiding with traditional encryption. To further reduce the running time, PwLMµ applies a chatic system with enhanced dynamics by µ [Chowdhury et al., 2024]. Zhao et al. proposed a usability enhanced scheme, which utilizes SPE to encrypt the lowest seven bits while adjusting the protected thumbnail and storing extra information in the most significant bit of each pixel. While the existing works mentioned above perform well in preserving the original thumbnail, they all generate the protected images with noticeable noise.

3 Proposed Scheme

In HIPP, We aim to protect the privacy of image details from both malicious users and their analysis models while retaining the overall visual effect of the original image to ensure the protected image remains natural. Here, image details means any information within thumbnail blocks such as facial feature and building appearance. As images shared in SNP are often displayed in thumbnail form for coarse-grained content presentation, HIPP reconstructs image details for privacy protection, and preserves the thumbnail as unchanged as possible for image contour preservation.

3.1 Overview

Figure 2 illustrates the overall framework of the proposed HIPP, which consists of two primary modules: the latent-toimage space mapper (G and G-1 ) and the detail information extractor E. Specifically, the n original images are transformed into the latent vectors {a1, · · · , an} by G-1 . And then, E divides the vector ai into two components: ath i and acot i , encapsulating the detail and contour information of the image xi, respectively. A random sequence {k1, · · · , kn} is generated by K and utilized to permute acot i , where ki ∈ [1, n] and ki ̸= kj (i, j ∈ [1, n], i ̸= j). The permuted detail vector acot ki is combined with ath i to form the new latent vectors a′i . Finally, the protected image yi is obtained by applying the generator G with a′i as input. The protected image yi has a thumbnail similar to xi, but the details are different. The backward section in Figure 2 represents the inverse process of HIPP, where k′ki = i, meaning the order of {a′1 , · · · , a′n } is exactly the same as that of {bk′1 , · · · , bk′n }. The restored image x′i is basically consistent with the original image xi. Additionally, maintaining the correct order of the protected images is crucial for successful restoration. This order can be securely encoded into image filenames or embedded within the protected images themselves via data hiding methods.

3.2 Latent-to-Image Space Mapper

To extract detail information from images in the latent space, a space mapper, consisting of an image generator G(·) : Z →

I and its inverse G-1 (·) : I → Z, is necessary to establish a mapping between the latent and image spaces. The latent vector corresponding to the original image x is obtained by

the reconstructed image ˜x is acquired by

and the latent vector corresponding to ˜x is derived by

Since the protected image y is expected to be restored to a version as similar to the original image x as possible, it is desirable to minimize the difference between a and ˜a as well as between x and ˜x.

In HIPP, we choose Glow [Kingma and Dhariwal, 2018] and Generative Adversarial Network (GAN) for image generation, each with its own strengths. For Glow, the whole processing procedure is completely reversible, enabling x and ˜x, as well as a and ˜a, to be perfectly consistent without any loss. For GAN, we apply StyleGAN [Karras et al., 2019] as G(·) and in-domain GAN inversion as G-1 [Zhu et al., 2020], which cannot achieve full reversibility but is capable of generating higher-quality protected images. The specific comparison results between Glow and GAN are presented in detail in the experimental section.

3.3 Detail Information Extractor

The detail information extractor E is crucial for both the generation of the protected image and the restoration of the original image, which aims to decouple image detail information from contour information within the latent vector a by

Here, acot is the detail vector and ath denotes the contour vector, both having the same dimensions as a. To specifically demonstrate the role of E in HIPP, we assume that we have the original images x1 and x2 now, with their corresponding latent vectors a1 and a2, respectively. After inputting a1 and a2 into E, we obtain acot 1 , ath 1 , acot 2 , and ath 2 . Then, two new vectors are derived by

which are subsequently inputted into G to generate the protected images y1 and y2. a′1 contains contour information from a1 and detail information from a2. Consequently, the generated protected image y1 has a contour similar to x1, but different details. In the backward process, we only need to perform a decoupling operation on the corresponding latent vector b1 of y1, which is identical to a′1 under Glow or approximately equal to a′1 under GAN:

Here, bcot 1 and bth 1 are basically the same as acot 2 and ath 1 , respectively. Therefore, we can obtain b′ 1 (which is essentially consistent with a1) by

and get the restored image x′1 by

As a, the latent vector mapped from x by Glow, is composed of multiple sub-vectors [z1, · · · , zl] (l denotes the scale of flow in Glow) of different dimensions, it is necessary to perform detail information extraction operations on each subvector separately as illustrated in Figure 3. First, z1 with size of wi × hi ××ci is reshaped into z′1 of size wihi × ci. z′′ i is generated by

where Linear(·) represents a full connected layer with identical input and output dimensions. zcot i is obtained through reshaping z′′ i , and then, zth i is derived through

For GAN, the vector a has size of 2 log2 h2 ×512 and reshaped into size of 1024 log2 h2 before Linear(·) operation. Overall, E aims to decouple a into acot and ath without changing the dimensions.

3.4 Loss Function

The ultimate goal of the proposed scheme is to train the detail information extractor E to decouple the detail and contour information within a as much as possible under frozen space mapper. The specific targets are: 1) the generated protected image y is expected to have a contour similar to the original image x; 2) y has details different from x; 3) the restored version of the protected image x′ is as close to x as possible. Therefore, the optimization objectives of HIPP are

Here, Thumb(x, b) sets the thumbnail block size to b × b and returns a thumbnail tx of x with the same size. More precisely, each pixel in the thumbnail is calculated by

where i ∈ [1,w], j ∈ [1, h], p ∈ [1, c], α = ⌈ ib ⌉ + 1, and β = ⌈ jb ⌉ + 1. And Dif(x, y) is utilized to evaluate the difference between x and y. A smaller Dif value indicates greater similarity and less difference between the two inputted images. However, E cannot be directly trained with the above optimization objectives since it operates in the latent space Z. The objectives must be transformed into loss functions related to the latent vectors. During training, a Gaussian-sampled vector r with the same dimension as a is used, and the loss functions shown in Figure 2 are introduced as follows.

1) Detail and Contour Loss: To achieve the first and second objectives, it is necessary to extract as much detail information from the vector as possible without including contour information. As x and its corresponding thumbnail share the same contour information, we minimize

to ensure that ath contains as little detail information as possible, where a = G-1 (x) and ath = G-1 (Thumb(x)). Here, MSE(x, y) returns the Mean Squared Error (MSE) of the inputs, which measures the difference between them. The formula for MSE is

(14)

(15)

Additionally, Ldis, calculated by

is minimized to ensure that ath contains sufficient contour information. Here, a′ = rcot + ath . Overall, acot is expected to retain as much detail information as possible while reducing the likelihood of containing contour information by simultaneously minimizing losses Lth and Ldis.

2) Reconstruction Loss: For the third objective, it is necessary to make the contour and detail vectors as independent of each other as possible so that E is able to extract the same

detail vector from a and r′ = acot + rth . Therefore, the loss

is minimized to enhance the reconstruction ability, ensuring that a′cot and r′cot extracted during the backward process are as close as possible to rcot and acot during the forward process, respectively. To this end,

is basically the same as a, making x as similar to x′ as possible.

Finally, the whole loss function is calculated by

where λ1 and λ2 are hyperparameters used to adjust the relative weights of the loss terms Ldis and Lrec in the overall loss function L. The optimization objective of HIPP is to minimize the whole loss function L.

4 Experiments

4.1 Experimental Setup

Datasets. In the experiments, 50000 images are randomly sampled from CelebA [Liu et al., 2015], Helen [Le et al., 2012], and LSUN[Yu et al., 2015] datasets and resized to 128 × 128 and 256 × 256 to form the training set. Additionally, we select 2000 images from the remaining images to

form a testing set.

Implementation Details. During the training procedure of E, we directly apply the pretrained G and G-1 models, keeping their parameters frozen. Meanwhile, an Adam optimizer with β1 = 0, β2 = 0.99, ϵ = 10-8 is applied, the learning rate and iteration are set to 10-5 and 200000, respectively. As for hyperparameters λ1 and λ2, they are set to 10 and 200, respectively, when applying Glow as G. If utilizing StyleGAN as G and in-domain GAN inversion as G-1 , the values of λ1 and λ2 are changed into 10 and 1000, respectively. In addition, when n is set to 1, a random sampled latent vector r is utilized to assist in transforming a into a′ as described in the loss function section. When n > 1, the specific process is illustrated in Figure 2.

Evaluation Metrics. There are three indicators utilized in experiments to assess the performance of HIPP: Peak Signalto-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [Wang et al., 2004], and Learned Perceptual Image Patch Similarity (LPIPS) [C.-V.Yang et al., 2014]. Specifically, PSNR quantifies the difference in intensity between the two images from a pixel-wise perspective. The higher the PSNR value, the smaller the difference between images. SSIM and LPIPS are both perceptual metrics that quantify the similarity between the two images. The SSIM value ranges from 0 to 1, with higher values indicating greater similarity. Conversely, the LPIPS value operates oppositely, where lower values signify higher similarity between the images.

Baselines. To comprehensively evaluate the performance of the proposed scheme, we select ten baselines for comparison: TP-JPEG [Wright et al., 2015], ATPE [Marohn et al., 2017], BIPU [Tajik et al., 2019], TPE-GAN [Chai et al., 2022], F-TPE [Zhang et al., 2022c], HF-TPE [Zhang et al., 2022b], FVPP [Zhang et al., 2023b], UE-JPEG [Ye et al., 2023], PwLMµ [Chowdhury et al., 2024], and PR3 [Zhao et al., 2024].

4.2 Experimental Results

Evaluation of Detail Privacy Protection

The privacy objective of HIPP is reconstructing details within each thumbnail block to maximize the inconsistency between the protected and original images. In the specific evaluation experiments, we extract facial feature vectors from the two images and then calculate the Euclidean distance between these vectors by d(p, q) = √∑i (qi -pi)2 , where pi and qi denote the coordinates of points p and q on the i-th dimension, respectively. As shown by the red section in Figure 5, the distance between face feature vectors of the images remains at a high level (the Eucldean distance between two unrelated facial vectors is approximately 0.836) regardless of the value of n. Furthermore, we apply a tool named face recognition 1 to judge whether faces in the input images belong to the same person. As shown by the blue section in Figure 5, the facial recognition accuracy of the protected images is quite low, which means that the proposed HIPP successfully reconstructs details in the original image. Meanwhile, as shown in Figure 6, the protected image y is not similar to the original image x from a perceptual perspective, indicating that unrelated details are successfully reconstructed by our scheme.

Evaluation of Image Contour Preservation

Figure 4 provides a visual demonstration of HIPP. It is obvious that the protected images generated by HIPP shown in the second row remains a natural version without visible noise. Meanwhile, thumbnails of the original and protected images are quite similar as displayed in the third and fourth rows. In addition, we apply SSIM and LPIPS to evaluate the similarity between thumbnails of the protected and original images, as displayed in Figure 6 and Table 1. The SSIM value reaches its peak when the block size is 64 × 64 under HIPP-GAN, which is higher than that in other approximately thumbnailpreserving works. Although the protected thumbnail under HIPP cannot be made entirely identical to the original one, it is sufficiently similar for users to associate it with the original image during online browsing. Figure 6 indicates that the similarity value reaches its highest when n = 1 and shows no significant change as n increases. Overall, the protected images generated by HIPP not only preserve the general contour of the original ones but also appear natural and are indistinguishable from realistic images.

1 https://github.com/ageitgey/face recognition

Evaluation of Reversibility

Although HIPP cannot fully restore the protected images into their original versions, the restored images exhibit a very high similarity to the original ones as shown in Figure 4 and 6. As in-domain GAN inversion cannot perfectly map the latent vector to the image without any loss, we introduce a new comparison object ˜x = G (G-1 (x)) to isolate the impact of the latent-to-image mapper on the evaluation of E. Additionally, we evaluate the quality of the restored images by face recognition with a threshold set to 0.3 as shown in Figure 5. Even under stringent threshold, the probability of identifying the original and restored images as belonging to the same person remains very high. Moreover, as shown in Table 4, the restored image under HIPP-Glow is almost identical to the original version. For HIPP-GAN, the results are less satisfactory due to the inherent non-inevitability of GAN. However, employing a more outstanding GAN inversion as G-1 can improve the restoration quality.

Ablation Study

To analyze the indispensability of each component in L, an ablation study is conducted by retraining E on different loss functions as shown in Table 2. Here, the final goal of training process is enlarging values of SSIM(tx, ty), SSIM(x, x′ ), SSIM(t˜x, ty), and SSIM(˜x, x) while reducing values of SSIM(x, y) and SSIM(˜x, y). Specifically, Lth becomes 0 when acot = a and ath = 0, Ldis = MSE(a′ , a) =

MSE(acot + rth , acot + ath ) turns to 0 whenacot = 0 and ath = a, and Lrec changes to 0 when E(a) = a, 0 or E(a) = 0, a. Results show in Table 2 indicate that HIPP successfully achieves the training objectives if and only if the loss function is set to L = Lth + Ldis + Lrec.

Efficiency Comparison

To evaluate the running time in detail, we process 1000 images at once and record the average running time per image as shown in Table 3. Compared to previous works, HIPP achieves a significant improvement in running time, saving processing time for users and enhancing its practicality. Meanwhile, the size expansion rate is calculated by comparing the file size of the protected image with that of the original image. Due to the absence of visible noise in the protected image, the images can be effectively compressed, resulting in a much smaller file size compared to other works. This not only improves the image uploading speed but also saves storage space.

5 Conclusion

In this paper, we propose a reversible image privacy protection scheme by preserving thumbnails, named HIPP, which can generate the protected image without any visible noise. The scheme maps the original image to the corresponding latent vector by the inversion of a generation model, decouples the vector into detail and contour vectors, and replaces the detail vector to reconstruct image details. Experimental results indicate that the generated protected image appears natural and has different details compared to the original version while preserving the whole contour unchanged. Furthermore, HIPP significantly reduces both runtime and file size of the protected image, thereby decreasing the time required for local processing and uploading, which enhances the practicability. In future work, we will explore ways to further decouple the image detail information from the contour information, aiming to achieve a higher level of privacy protection.

Acknowledgements

This work was supported by National Natural Science Foundation of China (NSFC) under Grants No, 62372334, No. 62202340, the Fundamental Research Funds for the Central Universities under No. 2042025kf0054, the Natural Science Foundation of Hubei Province under No. 2025AFB455, the CCF-NSFOCUS 'Kunpeng' Research Fund under No. CCFNSFOCUS 2023005, the Open Foundation of Henan Key Laboratory of Cyberspace Situation Awareness under No. HNTS2022004.