Prompt-Free Conditional Diffusion for Multi-object Image Augmentation

1 Introduction

In the past decade, deep neural networks have achieved a surge of success in a wide range of computer vision tasks [He et al., 2016; Dosovitskiy et al., 2020; Radford et al., 2021]. One key premise for such success lies on the collection of large-scale training images. However, in real scenarios even for a specific single task, amassing sufficient images to establish a dataset is often prohibitively costly and laboriously time-intensive, e.g., imageNet [Deng et al., 2009] for image classification. For this problem, a promising solution proves to be image augmentation with generative models [Antoniou et al., 2017], which aims at randomly generating extensive synthetic images based on a few manually collected images to rapidly establish a dataset. Following this idea, various effective image generative models [He et al., 2023; Chen et al., 2023] have been proposed successively. Among them, profiting from the powerful generative capacities, diffusion models have been paid increasing attention to image generation and augmentation. Usually, given some prompts related to the scene content, the diffusion model can directly generate a high-quality image with such a content.

† Corresponding author.

In real-world applications, generating multi-object images with complex spatial relationships is crucial. Although some recent progress have been made for multi-object image generation, due to much increased generation difficulty, most of these methods suffer from obvious limitations. As shown in Fig. 1, several existing methods [Wu et al., 2023b; Nguyen et al., 2023] use category names or image captions as conditions to inputs into the pre-trained diffusion model to generate images, and use attention maps to extract image labels. However, it is difficult to generate a large number of objects using only text prompts, and the quality of labels generated using attention maps is poor when objects overlap. Although some methods [Wang et al., 2024; Wu et al., 2023a] use stronger guidance, such as layout or paragraph, to improve the quality of generated images, these methods are difficult to scale. Some methods [Zhao et al., 2023; Xie et al., 2023] solve this problem by decomposing the multi-object image generation task. They first generate single-object images and their corresponding labels through a pre-trained diffusion model, and then use data augmentation to synthesize multi-object images. Although the number of objects and label quality are increased, artificial facts are often generated, which reduces the reality of the image.

In addition, the above methods tend to pursue trainingfree and directly use simple text prompts containing category names to generate data, which leads to deviations in style, size, etc. between the generated images and the original data. Furthermore, some methods [Suri et al., 2024; Yang et al., 2024] try to replace or add objects to the original images through image editing. Although this makes the augementated image as realistic as possible, the amount of information added is limited because the layout, background and most of the objects in the image have not changed.

To fill this gap, we propose a prompt-free conditional diffusion framework, aims to reduce the category and quantity deviations from the original data while improving the diversity of generated images. Inspired by image variation task [Ramesh et al., 2022; Xu et al., 2023; Xu et al., 2024], our framework utilizes a single multi-object image instead of text prompts as the condition of the diffusion model to reduce the category bias brought by text descriptions. More importantly, to better extract and inject the multi-object information into the diffusion procedure, we propose a local-global semantic fusion strategy that utilizes the pre-trained CLIP [Radford et al., 2021] model to separate extract the semantic knowledge within the whole condition image as well as its local crop. On the other hand, to further control the object amount as well as the layout diversity in the generated image, we further propose a reward model based counting loss to explicitly restrict the amount of objects in each category in the generated image, while imposing no any constraint on their spatial layout. By doing this, the proposed model is able to randomly generate high-quality images with the same number of objects in each category as the condition image or even more but showing different layouts, thus guaranteeing the variety of image augmentation. Experimental results demonstrate the superiority of the proposed method over several representative state-of-the-art baselines and showcase good downstream task gains and out-of-domain generalization capabilities.

In summary, this study mainly contributes in four aspects:

1. We propose a prompt-free conditional diffusion framework for multi-object image augmentation. By changing the text condition to a novel local-global semantic fusion strategy, which enables appropriate extracting the multiobject information from the condition image and injecting it into the diffusion model for image generation.
2. We design a reward model based counting loss to constrain the number of objects in each category of generated images, which improves the diversity of images.
3. We contribute new state-of-the-art performance of both downstream tasks and generated quality on MS-COCO dataset in terms of multi-object image augmentation.

2 RelatedWork

2.1 Text-to-Image DiffusionModels

Driven by multi-modal technology, text-to-image diffusion models exhibit formidable capabilities in image generation. GLIDE [Nichol et al., 2022] uses a cascade architecture and classifier-free guidance [Ho and Salimans, 2022] for image generation based on pre-trained language models. DALL-E2 [Ramesh et al., 2022] adopts a multi-stage model, using CLIP [Radford et al., 2021] text encoder to encode text and images. Imagen [Saharia et al., 2022] uses multiple text encoders to improve sample fidelity and text-image alignment. The latent diffusion model [Rombach et al., 2022] significantly reduces computational overhead by transferring the diffusion process from the image to a low-dimensional feature space. On its basis, exemplar-based methods [Li et al., 2024] achieve refined control of generated images under the guidance of text by introducing structural information as input, such as mask, edge, pose, etc. Subject-driven image generation methods [Ruiz et al., 2023; Gal et al., 2022] realize the customized generation of specific objects under the guidance of several target images and relevant text prompts. In contrast to the above techniques, the goal of our framework does not require specifying locations or customization of individuals for each instance, but to generate factual images with comparable object amounts and diverse layouts.

2.2 Image Variation

Given an image, image variation aims to generate an image with similar styles or semantics. Currently, there is no unified paradigm for image variation tasks. DALL-E2 [Ramesh et al., 2022] uses the alignment characteristics of the CLIP image and text encoder to encode input images to achieve image variation. ControlNet [Zhang et al., 2023] controls the generation of the diffusion model by adding an additional network structure based on the latent diffusion model. Its reference-only version achieves variation images by splicing the original attention layer of the diffusion model with the attention layer of the control network. Versatile Diffusion [Xu et al., 2023] designs a multi-stream multimodal latent diffusion model framework and supports the diversified generation of a single image stream. Prompt-Free Diffusion [Xu et al., 2024] replaces the text encoder with a semantic context encoder to learn the features of the input image and diversify it. Compared with the method proposed in this paper, the above method performs diversification on the entire image, and its diversified connotation often includes multiple information such as content, style, and color, which cannot guarantee that the instance of the generated image is consistent with the original image.

3 Methodology

3.1 ProblemFormulation

Despite the availability of excellent annotation tools such as SAM 2 [Ravi et al., 2024] and Grounding DINO [Liu et al., 2024], the diversified generation of large-scale multi-object images remains a problem that needs to be solved. In multiobject dataset augmentation, consider a collection of N samples, denoted as D = {(xi, yi), i = 1, ...,N}, where xi = {(oj, cj), j = 1, ...,N c i }, represents an input image containing N c i categories, and for each category cj, it contains oj objects. yi = {(bk, ck), k = 1, ...,N o i } denotes the category ck and the structured box annotations bk of N o i objects, and ∑N c i j=1 oj = N o i . The goal of the task is to generate a set of enhanced images D∗ = {x∗i , i = 1, ..., N} with the same number of input samples, where x∗i = {(ol, cl), l = 1, ...,N c∗ i }, requiring that for each category cj in the input image xi, a cl can be found in the augmented image x∗i corresponding to it, and the count of it ol ≥ oj.

3.2 Overall Architecture

As shown in Fig. 2, the proposed framework consists of two parts: a local-global semantic fusion strategy and a reward model based counting loss.

During the forward diffusion process, the pre-trained latent diffusion model first uses the encoder E to compress the input image x0i = xi ∈ RH×W×3 into a latent representation z0 i ∈ Rh×w×d , while the decoder D can transform the latent representation into pixel space, i.e., D(z0 i ) ≈ x0i , where Hh = Ww = 8 and d = 4. Then the noise ϵt is sampled from the Gaussian distribution and added to it, where t is a time step sampled from the uniform distribution. Finally, a DDPM is trained in the latent space based on the image condition pimg i using the MSE loss and our proposed counting loss to recover z0 i from the Gaussian distribution, where the MSE loss LMSE i is:

where zt i is the noise feature of time step t, ϵθ is the noise prediction network, which takes zt i as input and predicts the sampled Gaussian noise guided by the time step t and the conditional feature C(pimg i ), where C is our proposed localglobal semantic fusion module.

For the reverse diffusion process, the model directly samples noise in the latent space and uses the trained noise prediction network to gradually denoise it according to the conditional features to obtain the final image.

3.3 Local-Global Semantic Fusion

Numerous papers [Binyamin et al., 2024; Wen et al., 2023; Battash et al., 2024] point out that the text-to-image diffusion model often fails to generate images that accurately match the text prompt, especially when the prompt contains information such as multiple categories or counts. In addition, since most current multi-object generation methods pursue training-free and directly use text prompts to generate images, the generated category distribution is offset from the target dataset distribution due to the inherent bias of the generation model. To address the challenges of category bias introduced by textbased prompts, we replace textual prompts with image-based conditions for diffusion models. Using images as input conditions better captures the category distribution of the target dataset, reducing deviations and improving the fidelity of the generated data.

In order to adapt the latent diffusion model from textguided image generation to image-guided image generation, we use the image encoder Eimg pre-trained together with the original text encoder Etext using paired text-image data to encode the image condition, so that the obtained conditional features remain in the same feature space without fine-tuning all the parameters of the diffusion model.

The original text encoder uses hidden states of text conditions to capture the semantic relationship between the text context:

where C(ptext) ∈ Rbs×seq×emb is the output conditional feature, bs is the batch size of text condition ptext, seq is the sequence length, emb is the feature dimension, and T is the tokenizer. In order to further clarify the instance that needs to be enhanced, we crop it from the image, merge it with the original image and input it into the image encoder to extract

Algorithm 1 Counting Loss

Input: denoised image x∗i , open vocabulary object detector DOV , number of categories N c i , text prompt Si, class count list Lcount i , class index list Lindex i , counting loss step γ, counting loss threshold τ

1: if training steps larger than γ then 2: Let logitsi ←DOV (x∗i , Si). 3: for j ← 1, ..., N c i do 4: if Lindex i [j] is an integer then 5: Let sji ← logitsi[Lindex i [j]] 6: else 7: for idx in Lindex i [j] do 8: Let sji ←concatenate all logitsi[idx] 9: end for 10: end if 11: Calculate Lji use Eq. 6 12: end for 13: Calculate LCi use Eq. 7 14: return LCi 15: end if

features:

where pimg i is the local-global semantic fusion condition, and Crop(·) uses the bounding box information bi of the global image xi to crop the local instance to be augmented. In order to better understand the image context, we use hyperparameter pad to control the pixel of outward cropping.

Through the above operations, we express the information that is difficult to control with text, such as count and category, through batched local-global image information, highlighting its importance in the condition. To reduce computational complexity, we only need all the features of the original image, and for each cropped image, we only need its [CLS] feature:

where C(pimg i ) ∈ Rbs×(1+M)×emb , P is the image processor that processes the image condition for batch training. Specifically, for each image condition pimg i , the image processor fixes its cropped instances to M. When the number of instances is less than M, it is expanded with zero tensors, otherwise M instances are randomly selected for training. We set M to 9, which significantly reduces the computational complexity compared to the text condition while ensuring the semantics of most objects.

3.4 RewardModel Based Counting Loss

With the proposed local-global semantic fusion strategy, we can improve the fidelity of the generated image. To further ensure that the object amounts do not degrade, we propose a reward model based counting loss. Specifically, for the input image xi, we obtain the image x∗i by one-step denoising during training:

where ϵt is the noise prediction network, αt, αt, σt are the hyperparameters defined by DDPM, and z ∼ N(0, 1) is used to adjust the signal-to-noise ratio.

Then we use the image annotations to construct supervision information. The proposed counting loss focuses solely on category counts, rather than bounding box positions, to promote diverse layout generation. This design ensures that object counts match the desired distribution without imposing rigid spatial constraints, enhancing both flexibility and diversity in the generated images. For the N c i categories contained in the image, we first count the number of objects in each category and obtain a one-to-one corresponding category name list Lclass i = {name(cj), j = 1, ...,N c i } and count list Lcount i = {len(oj), j = 1, ...,N c i }, where name(·) is used to get the category name, and len(·) is a function of counting numbers. Then we connect each name in Lclass i with a period to construct the text prompt Si of the reward model. Since some category names have more than one word, we also record the index list Lindex i of each category in Si to obtain the result of the reward model.

Finally, we use the pre-trained open vocabulary object detector as the reward model to detect the categories in the image according to Si. For detection result of cj, we take the highest confidence sample based on input image and calculate the loss according to the threshold hyperparameter τ:

where sji is the confidence result of in category cj of image xi detected by the reward model. And the final counting loss LCi of image xi is:

The hyperparameter γ determines the training step at which counting loss begins to take effect. This avoids noisy gradients during early training stages when the denoised images may still contain significant noise. The calculation method of counting loss is outlined in Algorithm 1. The overall training loss of the proposed framework can be formulated as:

where λ is a hyperparameter for adjusting the loss weight.

4 Experiments

4.1 Experimental Setups

Datasets

We validate the proposed framework and comparison methods on the MS-COCO [Lin et al., 2014] dataset, a relatively complex object detection dataset containing 80 categories, with an average of 7.7 objects per image. We use train2017 containing 118K images to train the proposed method and generate images for downstream task evaluation, and use the COCO validation set val2017 consisting of 5K images for generation quality evaluation.

Implementation Details

We use Stable Diffusion XL [Podell et al., 2023] and Grounding DINO [Liu et al., 2024] as LDM and reward model respectively. We fine-tune the model using LoRA [Hu et al., 2021] at 512 × 512 resolution, we set the learning rate to 1e4, total batch size to 32, and train on two RTX 3090 GPUs using the AdamW [Loshchilov and Hutter, 2019] optimizer with constant scheduler. For training images, we use center crop and random flip as data augmentation. In the inference stage, we use the Euler scheduler with 50 steps for generation.

Metrics

In addition to the qualitative results, we use multiple quantitative indicators to evaluate our proposed method from various dimensions. For downstream task evaluation, we use mAP (mean Average Precision) and AP50 to evaluate the generated data, and for generation quality evaluation, we use the widely used Frechet Inception Distance (FID) [Heusel et al., 2017] to evaluate the fidelity of the generated images. In addition, to evaluate the diversity of the generated images, we calculate the diversity score (DS) by comparing the LPIPS [Zhang et al., 2018] metric of paired images. Finally, to evaluate the object amounts of the generated images, we designed an instance quantity score (IQS) that detects the instance quantity of each category under multiple confidence settings using the pre-trained YOLOv8m [Jocher et al., 2023] and compares it with the original images. Algorithm is shown in Appendix.

4.2 ComparisonMethods

We compare the proposed method with the state-of-the-art multi-object image augmentation methods Dataset Diffusion [Nguyen et al., 2023], Mosaic Fusion [Xie et al., 2023], Add SD [Yang et al., 2024], and image variation methods ControlNet Reference Only [Zhang et al., 2023], Versatile Diffusion [Xu et al., 2023], and Prompt-Free Diffusion [Xu et al., 2024]. We use the pre-trained model of the above methods to generate images under its default parameters.

4.3 DownstreamTask Evaluation

To verify the effectiveness of the proposed method, we use the generated data to train downstream detection and segmentation models and compare the metrics of the validation set to test the ability of image augmentation. Specifically, we use the training set of the COCO dataset to generate 10k data for each method and mix it with the original training set to train Mask RCNN[He et al., 2017]. For Dataset Diffusion, we use the pre-trained model to perform instance segmentation on its semantic labels. For Add SD, image variation methods, and our method, we use Grounding DINO[Liu et al., 2024] and SAM[Kirillov et al., 2023] to generate annotations.

Tab. 1 shows the performance indicators of all methods on the validation set. Our method achieves the state-of-the-art performance on both detection models. These results demonstrate the effectiveness of the proposed method and provide promising results for the further application of generative models in detection tasks.

4.4 Generation Quality Evaluation

To further verify the generation quality of the model, we use the validation set of the COCO dataset to evaluate the generated images. Specifically, we use each image in the validation set as a condition for image augmentation and calculate the fidelity, diversity score, and instance quantity score of all images. Since Dataset Diffusion [Nguyen et al., 2023] and MosaciFusion [Xie et al., 2023] do not support image-based augmentation, we do not compare with them here.

Qualitative Evaluation

Fig. 3 shows the visualization results on some challenging samples. Add SD[Yang et al., 2024] does not always successfully add targets. ControlNet[Zhang et al., 2023] cannot understand the semantics of the input image and loses the object of interest after image augmentation. The diversity of the images generated by Versatile Diffusion[Xu et al., 2023] and Prompt Free Diffusion[Xu et al., 2024] is not good. The layout of the image is almost the same as the original image, and there will be problems with missing targets and even counterfactual images. Compared with the above methods, our method achieves the best results in the balance of layout diversity, number of generated objects, and consistency with facts.

Quantitative Evaluation

As demonstrated in Tab. 2, val2017 represents the results of the original val dataset. The proposed method achieves the best or suboptimal results across FID, DS and IQS metrics, which proves the effectiveness of our method. It is worth noting that although Add SD [Yang et al., 2024] is superior to the proposed method in terms of fidelity through image editing, its diversity score is greatly reduced, and the instance quantity score is even lower than the original dataset. The proposed method has optimal performance in terms of the balance of fidelity, diversity and instance quantity.

Out-of-domain Evaluation

We also conducted experiments on the out-of-domain generalization ability of the model, including two settings. The cross-view setting is shown in the first two columns of Fig. 4. We experiment with satellite and drone remote sensing images. Our method can correctly understand the semantics in the input image and perform cross-view image augmentation on it. The cross-category setting is shown in the last two columns of Fig. 4. We test categories such as chickens, eggs, forklift, et al. that are not in the COCO dataset. Our method can understand the semantics of unseen categories through only images and generate diverse images.

Multiple Random Generation & Recurrent Generation

As shown in Fig. 5, we verified the effects of different methods on multiple augmentations of a single image and further augmentations of the augmented image. Our method achieved the best layout diversity. More results can be found in the supplementary materials.

4.5 Ablation Study

Effectiveness of Each Component

To further verify the effectiveness of our proposed components, we conducted a series of ablation studies. These studies mainly focus on two key components of our model: the local-global semantic fusion strategy and the reward model based counting loss. We use SDXL img2img as our baseline. As shown in Tab. 3, after using semantic fusion module to replace the original text condition module, the fidelity and diversity of the model have been significantly improved. Similarly, after adding counting loss, the instance quantity score of the model was further improved.

Analysis of Different Conditions

We also conducted ablation experiments under different conditions. As shown in Tab. 4, although both category name and content conditions can improve the performance of the model, the category name is a subset of the content, and using only the content can enable the model to achieve higher performance.

Analysis of Different Inference Methods

In practical applications, we cannot always obtain the ground truth of image annotations. So we use random crop and Grounding DINO detection results as contents for inference. As shown in Tab. 5, the model can achieve similar fidelity and diversity with a slight decrease in instance quantity score.

5 Conclusion

This paper introduces a prompt-free conditional diffusion framework for multi-object image augmentation. Through the proposed local-global semantic fusion strategy and the reward model based counting loss, the model can augment the images in a large-scale and diverse manner that conforms to the original category distribution. Qualitative and quantitative experimental evaluations substantiate the efficacy and superiority of our proposed methodology. At the same time, both out-of-domain generalization ability and recurrent augmentation ability of the model provide more possibilities for its application.

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China under Grand 62372379, Grant 62472359, and Grant 62472350; in part by the Xi'an's Key Industrial Chain Core Technology Breakthrough Project: AI Core Technology Breakthrough under Grand 23ZDCYJSGG0003-2023; in part by National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing fundation under Grant TJ-0423-04; in part by Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University under Grant CX2025092.