DiffStega: Towards Universal Training-Free Coverless Image Steganography with Diffusion Models

1 Introduction

The internet revolution has significantly facilitated communication, yet posing challenges in securing messages transmitted over the Internet [Mandal et al., 2022]. Steganography is a popular technique to hide information into the container in an imperceptible manner [Yu et al., 2024]. As a result, only trusted receivers are able to recover the information from the steganographic content. As a subset of this field, image steganography specializes in disguising the secret message as an image, offering a high degree of security and privacy. It has applications in diverse areas, including image compression [Jafari et al., 2013], secure communication [Duluta et al., 2017], and cloud computing [AlKhamese et al., 2019]. Traditional cover-based image steganography schemes hide the secret message in a cover image by altering its statistical properties [Meng et al., 2023]. Once the cover image is leaked, the hidden message can be easily detected by steganalysis [Karampidis et al., 2018]. In contrast, coverless image steganography (CIS) [Zhou et al., 2015] aims to encode or map the secret message into a stego image rather than modifying a cover image. Thus, it has greater imperceptibility compared to cover-based techniques.

∗Corresponding authors.

CRoSS [Yu et al., 2024] has demonstrated the potential of using diffusion models for CIS. This approach consists of two stages: hiding and recovery. In the hiding stage, text prompt 1, serving as a private key, guides the Denoising Diffusion Implicit Model (DDIM) [Song et al., 2021] inversion process to convert an original image to an initial noisy latent code. Subsequently, text prompt 2, acting as a public key, directs the DDIM denoising process to produce a stego image from this latent code. However, the reliance on text prompts as private keys is vulnerable. As illustrated in Figure 1, if Eve knows about what Alice might encrypt through a background investigation, he could easily infer the correct private prompt. In practice, even a similar private prompt may be sufficient to breach the encryption. Furthermore, since the private key is contingent on the content of the original image, it must be transmitted for each use, creating a potential risk of leakage.

We posit that a general CIS task should offer substantial flexibility, meaning that the images intended for concealment could be encrypted into various contents or styles. To this end, we categorize the general CIS task into two distinct types: content-based steganography and style-based steganography. Content-based steganography alters the primary object categories within an image while preserving its inherent structure. In contrast, style-based steganography achieves near-total imperceptibility by translating images into artistic renditions. This approach further obscures the detection and identification of hidden objects. Meanwhile, diffusion-based CIS should be able to produce stego images with arbitrary target text, especially general or similar texts.

To achieve general CIS, we introduce a novel diffusionbased pipeline. Akin to traditional cryptography, our method hinges on a specific numerical password, predetermined and known only to trusted parties. Since our password is independent of image contents, the password only needs to be transmitted once. Utilizing public text prompts and the prearranged password, trusted parties can generate a specific image to guide both the hiding stage and the recovery stage. To enhance the critical role of the password, we propose the Noise Flip technique, which encrypts the noisy latent code at the deepest step of the diffusion model while minimally altering its mean and variance. This innovation substantially complicates the task for potential attackers while simultaneously ensuring superior generation quality. To objectively assess our method's efficacy in general CIS tasks, we have curated a specialized dataset. This dataset comprises images alongside their target content, style, and analogous prompts. Our experimental results indicate significant advancements over existing methods, particularly in aspects of versatility, password sensitivity, and the effectiveness of recovery.

Our contributions can be summarized as follows:

• We overcome the limitations of current diffusion-based CIS models by incorporating pre-determined passwords, leveraging existing models without further fine-tuning.
• We introduce a novel CIS pipeline, which uniquely employs pre-determined passwords instead of text prompts alone to ensure security. This eliminates the need to transfer the private key each time the secret image is changed and is adaptable to any text prompt.
• We create a specialized dataset tailored for general CIS tasks. Comprehensive experiments show the superior

performance of ours in comparison to existing methods.

2 RelatedWork

2.1 Image Steganography

Cover-based Methods. Traditional cover-based methods are divided into spatial domain-based methods that directly modify the pixels of the cover image [Yang et al., 2008; Pevn`y et al., 2010], and transform domain-based methods that embed information into frequency domains [Chen, 2007; McKeon, 2007; Valandar et al., 2017]. Deep learning-based methods use neural networks to hide information in cover images[Baluja, 2017; Jia et al., 2022b; Jia et al., 2020; Jia et al., 2022a]. SteganoGAN [Zhang et al., 2019] uses generative adversarial networks to optimize image quality. HiNet [Jing et al., 2021] introduces invertible neural networks into steganography tasks. Changing the cover image leads to the usual drawbacks of cover-based methods with steganalysis [Karampidis et al., 2018]. Related topics include Image Forgery Detection [Guo et al., 2023] and invisible Watermarking [Fu et al., 2024] have also been studied extensively [Liu et al., 2022; Wu et al., 2023; Gao et al., 2015].

Coverless Methods. Coverless methods hide information without cover images. Early CIS refers to mapping the secret message into another image without modification [Meng et al., 2023]. Therefore, it fundamentally resists steganalysis and significantly improves security. Recently, CRoSS [Yu et al., 2024] uses diffusion models and the inversion technique to achieve CIS. It uses prompts as private and public keys to translate the secret image into another, which is more controllable and robust with high quality. Diffusion-based CIS is promising with the powerful generation ability and rapid development of diffusion models.

2.2 DiffusionModels

Diffusion models are the newly emerged generative models, which synthesize images via progressively denoising Gaussian noise. Among them, DALLE-3 [Betker et al., 2023], Imagen [Saharia et al., 2022] and Stable Diffusion [Rombach et al., 2022] have achieved the state-of-the-art results on many computer vision tasks.

Controlled Generation. Recent advances provide additional control for diffusion process. ControlNet [Zhang et al., 2023] and T2I-Adapter [Mou et al., 2023] leverage adapters to add conditional controls of semantic segmentation, OpenPose bone image, and Canny edge detection, etc. In addition, BLIP-Diffusion [Li et al., 2023] and IP-Adapter [Ye et al., 2023] handle reference images as additional prompts to control the generation with image embeddings.

Diffusion Inversion. Diffusion inversion deterministically noise the image to the intermediate latent code along the path that the denoising would follow with the same conditioning [Wallace et al., 2023], widely used for image editing tasks. Among them, DDIM [Song et al., 2021] relies on local linearization assumptions, suffering errors with the actual process. To reduce inversion errors, Null-text Inversion [Mokady et al., 2023] introduces additional fine-tuning on optimizing the null embedding, and EDICT [Wallace et al., 2023] uses coupled transformations without fine-tuning.

Figure 2: The pipeline of the DiffStega. (a) We use text prompts and Iref generated by RefGen with password Pcrt to guide the diffusion process of hiding stage. The text prompts and the optional control image Ictrl (e.g. OpenPose bone image) is set public. (b) With public resources, authenticated parties could reproduce the same Iref with Pcrt to guide the diffusion process of recovery stage with text prompts. (c) It illustrates the scenario where attackers attempt to directly recover the image without any password. (c) Wrong password Pwrg would result in wrong Iref , which is distinct from the correct reference image, resulting in misleading the recovery diffusion process. Green / Red denotes the correct / wrong decrypted items. For brevity, we omit the encoder and decoder of VAE for latent diffusion models.

3 Method

3.1 Overview

Our method consists of two stages: hiding and recovery. In hiding stage, shown in Figure 2a, we first generate a reference image Iref with given password Pcrt, prompt 2 and optional control image Ictrl (e.g. OpenPose bone image). Then we translate original secret image Iori into encrypted image Ienc with the guidance of prompt 1, prompt 2, and Iref . We set prompt 1, prompt 2 and Ictrl as public resources. In recovery stage, shown in Figure 2b, the secret image is recovered from Ienc with correct password Pcrt and public resources in a reverse procedure of hiding stage. If a malicious attacker attempts to directly recover the image without any passwords, or with a wrong password, a distinct recovery would be obtained, shown in Figure 2c and in Figure 2d respectively. To control the diffusion process in the two stages, we propose Reference Generator (RefGen) to generate Iref as image prompts, and Guidance Injection module as detailed in Section 3.2 and Section 3.3 respectively.

Hiding Stage

In hiding stage, DiffStega mainly uses two encryption methods. One is RefGen to generate Iref that guides the reverse diffusion process. Another is Noise Flip to ensure that the original image cannot be recovered with wrong passwords. We introduce the hiding stage as follows.

Preparation. We first input private password Pcrt, public text prompt 2, and optional control image Ictrl (e.g. OpenPose bone image, semantic image) into RefGen, to generate a reference image Iref . Note that we only set text prompts and Ictrl public. Pcrt and Iref will not be published publicly.

Forward Diffusion. Since we use lattent diffusions [Rombach et al., 2022], Iori is first encoded by VAE into latent codes x0. Then we use diffusion inversion to convert x0 to initial noisy latent code xT , where T is DDIM [Song et al., 2021] steps. Since we choose EDICT [Wallace et al., 2023] as noising inversion method, our method is training-free. This process requires only null-text prompt 1 as guidance.

Noise Flip. To improve the influence of Pcrt, we use it as random seed to deterministically flipping partial positions in xT , resulting in a slightly different x′T . We denote this procedure as Noise Flip. The more positions flipped, the harder for attackers to recover the original image without the correct password. This Noise Flip process follows the formula:

where ⊙ denotes element-wise multiplication, and Mrand is a random binary mask generated with deterministic random seed according to Pcrt. The proportion of 1 in Mrand is controlled through the coefficient η ∼ [0, 1]. Small η is sufficient for significant modification of xT . After this procedure, the noisy latent code of step T has strong dependence on Pcrt, while barely altering its mean and variance.

Reverse Diffusion. In this process, we use Guidance Injection to treat Iref as image prompt to guide noise prediction together with text prompt 2. Then, the reverse diffusion process converts x′T into x′0 with EDICT [Wallace et al., 2023] denoising. Finally, x′0 is decoded into encrypted image Ienc with VAE. With the aid of Iref and Noise Flip, Ienc would be distinct from Iori. Only authenticated users with Pcrt could recover the original image. For brevity, we use one latent code to represent coupled latent pairs used in EDICT.

Recovery Stage

As illustrated in Figure 2, the recovery stage is the reverse process of the hiding stage in symmetry. With private Pcrt, authenticated parties could easily recover the original image, where attackers is hard to predict the correct numerical password. In this stage, we have encrypted image Ienc and public resources consisting of prompt 1, prompt 2, and Ictrl. The possible scenarios are that we have the correct password Pcrt, or the wrong password Pwrg, or do not use any password at all. We will discuss the details of these scenarios as follows.

Recovery with Correct Password. With Pcrt, public prompt 2 and Ictrl, we first use RefGen to reproduce the correct Iref , identical to that used in hiding stage. Then we use Iref and prompt 2 to guide noise prediction together via Guidance Injection. VAE encodes Ienc into ˆx′0 . The forward diffusion process convert ˆx′0 into ˆx′T with EDICT noising inversion. Following the same procedure of Noise Flip in hiding stage, we could just flip back the previous reversed positions in ˆx′T , resulting in ˆxT ≈ xT . Then we use reverse denoising process via EDICT to convert ˆxT to ˆx0. After VAE decoder, we finally get correctly recovered image Irec.

Recoverywithout AnyPassword. Formalicious attackers, the simplest way they would try is directly conducting recovery with only public text prompts, as shown in Figure 2c. It removes the guidance of Iref and Noise Flip from the correct recovery procedure. However, without the guidance of Iref , there is a gap between ˆx′T and x′T . Moreover, since flipped positions in ˆx′T remain, ˆx0 is distinct from x0, leading to awfully wrong recovery.

Recovery withWrong Password. As shown in Figure 2d, wrong password Pwrg would produce wrong Iref , which is distinct from that used in hiding stage. It results in wrong ˆx′T from ˆx′0 in the forward diffusion process. After Noise Flip with a wrong random seed, ˆxT would be further far different from the original xT . Since prompt 1 is null-text, it is nearly impossible for the final Irec to present the content with the same semantics as the original image.

3.2 Reference Generator

In this section, we will introduce the inside procedure of RefGen. It aims to produce deterministic images according to the password and Ictrl. As shown in Figure 3, RefGen first generates a deterministic Gaussian noise with the random seed according to given password P. The Gaussian noise serves as the initial noisy latent code of generation. We use ControlNet [Zhang et al., 2023] to add additional control with Ictrl (e.g. OpenPose bone image) to diffusion models. Then the reference image Iref is obtained from pretrained diffusion models with the guidance of Ictrl and prompt 2.

3.3 Guidance Injection

This procedure aims to inject image features of Iref to the diffusion models. In order to use images to guide the diffusion process in the same way as the text, we adapt IP-Adapter [Ye et al., 2023] to inject image features into U-Net on the low dimensional latents. Furthermore, we notice that if we use optional control image Ictrl to force Iref have the similar structure with Iori, the encrypted image Ienc usually have the similar structure with the aid of Guidance Injection.

3.4 Security Guarantee

Previous work uses prompt 1 as private key and prompt 2 as public key [Yu et al., 2024]. They believe that attackers can only guess the prompt 1 by exhaustive method, being unable to judge which is the true Iori from candidate recovered images. However, we believe that potential attackers usually conduct many investigations on the target. For example, if the target is a military organization, the private key is likely to be related to weapons and equipment. Moreover, frequently transferring private prompt 1 faces risks. In order to make the steganography and recovery only depend on the specified password, DiffStega encrypts the original image with the guidance of images generated with passwords.

In the whole pipeline, Pcrt only needs to be transmitted once and is independent of the original image contents. For trusted parties, Iref can be losslessly reproduced through Pcrt and guides recovery stage to restore the original image. Null-text prompt 1 makes it impossible for an attacker to speculate on the original content of the image, which makes the wrong decryption less likely to be the original content. And our pipeline can resist steganalysis and many kinds of distortion because of the inherent advantages of diffusion models, which has been explained in [Yu et al., 2024].

4 Experiment

4.1 Implementation Details

For all experiments in this paper, we use the pre-trained SD v1.5 1 in the forward diffuion of hiding stage and the reverse diffusion of the recovery stage. And we use PicX real 2 in RefGen. We use SD v1.5 for experiments on style prompts in the reverse diffuion of hiding stage and the forward diffusion of the recovery stage, but PicX real for other experiments. We set T = 50, and the mixing coefficient of EDICT is 0.93. We

1 https://huggingface.co/runwayml/stable-diffusion-v1-5

2 https://huggingface.co/GraydientPlatformAPI/picx-real

1

Figure 4: The visual comparison of DiffStega and CRoSS families on UniStega dataset with different prompts. For DiffStega (ours), we categorize the recovery it into three possible scenarios. Besides recovering with correct private key, malicious attackers may attempt recovery without any password, ignoring the use of Guidance Injection and Noise Flip, or recovery with the wrong password Pwrg. Note that although we display prompt 1 bellow images, DiffStega still uses null-text as prompt 1 instead. CRoSS* uses two diffusion models consistent with DiffStega rather than a single model in CRoSS.

use IP-Adapter-plus [Ye et al., 2023] in Guidance Injection, and its weight factor is 1. The guidance scale of diffusion models is 1. η = 0.05 in Noise Flip. The diffusion process for ours is executed over steps [0, ξT ]. We set ξ = 0.7 for experiments on style prompts and ξ = 0.6 for other prompts. DiffStega uses ControlNet [Zhang et al., 2023] with additional control images in RefGen for all experiments except for style prompts. For additional control images, DiffStega uses semantic segmentations from OneFormer [Jain et al., 2023] as control images in general, but OpenPose bone images for face images. Experiments on style prompts use no control image. All experiments are conducted on single Nvidia RTX 3090 GPU, requiring no additional training or fine-tuning.

Since there is only one diffusion-based CIS model, we mainly compare DiffStega with CRoSS [Yu et al., 2024]. For fair comparison, CRoSS* is the modefied version which uses the same two models used in DiffStega, rather than only SD v1.5 in the original CRoSS. And all models use EDICT inversion. Also, CRoSS families use ControlNet with the same control images in the reverse diffuion of hiding stage and the forward diffusion of the recovery stage when DiffStega uses them. For sufficient inversion steps, ξ is set to 1 for CRoSS families. Meaningful description texts are only used as prompt 1, i.e. private keys, for CRoSS families. On the contrary, DiffStega uses null-text as prompt 1. All models use target texts as prompt 2. We also conduct security and robustness experiments compared with HiNet [Jing et al., 2021].

4.2 Datasets andMetrics

Datasets. Since our method supports universal CIS under arbitrary text prompts and can be applied to versatile images, we build a dataset UniStega, consisting of 3 subsets with a total of 100 images under different scenarios: (1) UniStegaContent comprises of 42 images with corresponding prompts and target content prompts. It applies to the most common CIS scenario with similar shape but different content; (2) UniStega-Style comprises of 28 images and prompt pairs. The target prompts refer to artworks in the styles of famous artists. It applies to CIS with higher difficulty to guess what the original images are; (3) UniStega-Similar comprises of 30 images and analogous prompt pairs. It applies to CIS with similar or overly general prompts. All images are from public dataset COCO [Lin et al., 2014], AFHQ [Choi et al., 2020], FFHQ [Karras et al., 2019], CelebA-HQ [Karras et al., 2018] and Internet, center cropped and resized to 512 × 512. We use BLIP [Li et al., 2022] to generate description prompts, and Llama2 [Touvron et al., 2023] to generate target content prompts with semantic modifications, or artificial adjustment to generate other prompts, following CRoSS [Yu et al., 2024].

Metrics. We use PSNR, SSIM [Wang et al., 2004], LPIPS [Zhang et al., 2018], and ID Cosine Similarity from Facenet [Schroff et al., 2015] for face images to assess the quality of hiding and recovery. We use CLIP Score [Radford et al., 2021] to evaluate whether the encrypted image matches the target text promt or not. We use NIQE [Mittal et al., 2012] to blindly assess the naturalness of encrypted images.

4.3 Experimental Results

Qualitative Results. Figure 4 shows the visual comparision of DiffStega and CRoSS families on UniStega dataset. We categorize it into three possible scenarios. The first is that authenticated users recover the original images with correct private key. Although DiffStega makes more modifications to the original images, it performs more accurate recovery. The others are that attackers attempt to recover without any password, i.e. not using RefGen and Noise Flip, or recovering with a wrong password. Attackers even hardly recover the correct category of original objects. For style prompts, the encrypted image of CRoSS loses too many details of original images, resulting in a significant difference between recovered images and the original. For similar prompts, using a single diffusion model, the encrypted images of CRoSS are almost the same as the original. CROSS* fails in recovery because of inversion errors and general prompt 1. However, with the guidance of reference image and fewer diffusion steps, DiffStega could still achieves satisfactory performance.

Quantitative Results. For encryption with steganography shown in Table 1. The smaller the similarity to the original

(a) Different controls.

(b) Different checkpoints.

Figure 6: The encrypted and recovered images of DiffStega with different controls (bottom right) and checkpoints of diffusion models.

image, the better the hiding performance. DiffStega makes the encrypted images distinct from the original ones. Due to the lack of more effective guidance, CRoSS has the worst encryption performance. Meanwhile, the CLIP Scores between encrypted images and prompt 2 shows that DiffStega has better consistency with target prompts. Moreover, DiffStega has stronger identity hiding capability of face images than CRoSS families. For recovery of different scenarios shown in Table 1. The greater the similarity to the original image, the better the recovery performance with correct private key. For scenarios of wrong recovery, the opposite is true. DiffStega shows better recovery performance than CRoSS families.

Imperceptibility and Security. Table 2 shows that ours has similar NIQE score to the original images, hardly suspected by people. For anti-analysis security, We use XuNet [Xu et al., 2016] to distinguish the encrypted images to general images without steganography. As shown in Figure 5, the closer the detection accuracy approximates 50%, the more secure the method is. DiffStega shares the similar performance with CRoSS families, much better than HiNet.

(b) Different Noise Flip scales η.

(c) Different diffusion steps with ξ.

Table 2: NIQE scores that indicate the quality of encrypted images. All methods is relatively as natural as the original images.

Robustness and Controllability. Real-ESRGAN [Wang et al., 2021] is used to perform nolinear image enhancement for degradations, denoted as Gaussian deblur and JPEG enhancer. As shown in Table 3, DiffStega has similar robustness to CRoSS, while HiNet suffers significant drops in PSNR. Figure 6 shows that our pipeline is flexible and applicable to different controls and checkpoints 3 .

4.4 Ablation Study

Influence of Null-text Prompt. As shown in Figure 7a, meaningful prompt 1 narrows the similarity between the wrong decrypted image and the original image. Considering that publicly disclosing meaningful prompt 1 is equivalent to letting everyone know what is encrypted, but only brings minimal performance improvement as shown in Table 1, we prefer to set null-text as prompt 1 for better security.

Influence of Noise Flip. As shown in Table 1 and Table 1 for DiffStega‡ , Noise Flip increases the reliance on the private password. Without Noise Flip, DiffStega has slight improvement in the quality of recovery, but double performance drop on encryption and wrong decryption. As η grows in Figure 7b, although wrong decrypted images are getting further away from the original, the quality of the encrypted images is getting worse and worse. Therefore η = 0.05 is sufficient. The second row demonstrates that different Noise Flip scales have limited influence of the correct recovery.

3 https://huggingface.co/stablediffusionapi/majicmix-fantasy

Influence of ξ. As shown in Figure 7c, DiffStega achieves distinct change and meets the target description when ξ ≥ 0.6. However, only when ξ is close to 1 can CRoSS families make the encrypted image distinct from the original. The red box indicates the values we recommended in the experiments.

5 Limitation

Because there is inevitable error in diffusion inversion methods, DiffStega suffers the poor recoverability if ξ is too large. Meanwhile, DiffStega introduces additional processes of generating reference images, which means more computational overhead. However, the additional cost will gradually be negligible with fast sampling methods [Luo et al., 2023].

6 Conclusion

This paper proposes DiffStega, cleverly designed for general coverless image steganography with diffusion models. We uses the pre-determined password as private key, and propose Noise Flip to achieve high quality steganography and nearly undistorted recovery of the original images. Extensive experiments show our superiority compared with previous methods. How to directly influence the image generation process with passwords is a promising research topic in the future.

Acknowledgments

This work was supported in part by the China Postdoctoral Science Foundation under Grant Number 2023TQ0212 and 2023M742298, in part by the Postdoctoral Fellowship Program of CPSF under Grant Number GZC20231618, in part by the National Natural Science Foundation of China under Grant 62301310 and 62301316, and in part by the Shanghai Pujiang Program under Grant 22PJ1406800.

Contribution Statement

Yiwei Yang and Zheyuan Liu contributed equally.