AK4Prompts: Aesthetics-driven Automatically Keywords-Ranking for Prompts in Text-To-Image Models

1 Introduction

Text-to-Image Synthesis (TIS) is a highly advanced and extensively utilized technique in generative Artificial Intelligence, aimed at generating realistic images based on textual input [Zhang et al., 2023]. Recently, with the advancement in the modeling capabilities of large models, TIS is undergoing a revolution. Cutting-edge text-to-image diffusion models like DALLE [Ramesh et al., 2021] and Latent Diffusion Models (LDMs), such as Stable Diffusion [Rombach et al., 2022], have emerged as pioneers in image generation, leveraging training data at a web scale. Additionally, Consistency models [Song et al., 2023], a new class of generative models capable of producing high-quality samples through a single network evaluation, have inspired further advancements. Building upon this concept, Luo et al. [Luo et al., 2023a] propose Latent Consistency Models (LCMs), which enable rapid inference with minimal steps on any pre-trained LDM and significantly enhance the speed of TIS. However, the image generation quality in these existing models relies heavily on the sophisticated design of keyword-based text prompts [Zhong et al., 2023]. This reliance stems from the training data quality, necessitating detailed prompts to produce high-quality images [Betker et al., 2023]. Determining the most effective keywords for a given prompt content often requires iterative experimentation and research within online communities [Oppenlaender, 2022]. In real-world scenarios, individuals lacking expertise often encounter challenges in selecting appropriate keywords for composing detailed prompts. This necessity leads to subjective assessments of the impact and quality of chosen keywords through repeated generation attempts, resulting in significant time and resource losses.

∗Corresponding author.

Prompt engineering is an emerging research field focused on developing more effective prompts for deep generative models. In the TIS community, a common practice involves investigating the effects of various keywords. BestPrompt [Pavlichenko and Ustalov, 2023] initially explores these effects by employing human evaluation to assess a limited number of text prompts and keywords, ultimately selecting a set of exceptional keywords. However, this approach has inherent limitations and inefficiencies, posing challenges in comprehensively and objectively exploring the impact of all keywords on generated results. Furthermore, image generation is jointly controlled by the interaction between keywords and basic prompt content. However, this study only considers the individual influence of keywords, neglecting potential variations in their effects across different prompt categories. Another prevalent approach involves fine-tuning a language model for TIS prompt generation using com-

Original Prompt: "vase of mixed flowers."

plex prompts datasets, such as MagicPrompt 1 and BeautifulPrompt [Cao et al., 2023]. However, these methods primarily focus on completing or rewriting prompts, potentially altering the content entities. Moreover, the generated prompts often lack flexibility and exhibit excessive rigidity.

In this paper, we propose a keyword evaluation model named AK4Prompts that can explicitly quantify the multidimensional impact of different keywords on the quality of generated images in response to user prompts. Our AK4Prompts automatically selects the keywords that best match the original prompt based on users' preferences, effectively enhancing the quality of the generated images. Our method preserves the content of the original prompt and allows users to customize their final outputs, thus overcoming the limitations of previous approaches. Meanwhile, it significantly alleviates the complexity of prompt design and the difficulty of selecting keywords, thereby saving users considerable trial and error time.

Figure 1 illustrates the evolution of images generated by setting different degrees of specific preferences. In summary, the main contributions of this study are as follows:

• We curate a high-quality set of keywords and develop an evaluation framework based on established TIS models. This framework enables automated and efficient assessment of these keywords, eliminating the need for manual intervention.
• We propose AK4Prompts, a keyword evaluation model that can explicitly quantify the multidimensional impact of different keywords on the quality of images generated by TIS models based on user prompts.

1 https://huggingface.co/Gustavosta/MagicPrompt-StableDiffusion

• Our approach demonstrates the judicious selection of the most appropriate keywords based on user prompts and preferences. Extensive experimental results showcase its superiority over strong baselines.

2 RelatedWork

2.1 Text-to-Image Synthesis (TIS)

TIS is a multi-modal task involving the generation of images based on textual inputs. In the early years, popular image generation tasks were mainly based on Generative Adversarial Network (GAN) [Goodfellow et al., 2014; Reed et al., 2016]. More recently, diffusion models [Ho et al., 2020; Song et al., 2020; Sohl-Dickstein et al., 2015; Dhariwal and Nichol, 2021] have achieved remarkable results. Subsequently, LDMs (e.g., Stable Diffusion [Rombach et al., 2022]) apply a diffusion model in the latent space using a pre-trained autoencoder for efficient training and inference. This development has resulted in the emergence of high-performance generative models trained on billions of data points, including DALLE-2 [Ramesh et al., 2022], Imagen [Saharia et al., 2022], Imagic [Kawar et al., 2023], and SDXL [Podell et al., 2023]. Among these models, the Stable Diffusion series stands out as open-source models with an active user community.

Recently, LCMs [Luo et al., 2023a], LCM-LoRA [Luo et al., 2023b], and SDXL-Turbo [Sauer et al., 2023] have emerged as solutions to the slow sampling issue in image generation, inspired by Consistency Models [Song et al., 2023]. LCM-LoRA serves as a universal, training-free acceleration module. It can be directly integrated into various Stable Diffusion fine-tuned models or LoRAs [Hu et al., 2021], facilitating fast inference with minimal steps. This allows us to efficiently infer and explore a large number of keyword and prompt category combinations.

However, the qualities of generated images are dependent on the keywords provided in the prompts. TIS models usually utilize a text encoder, which employs pre-trained language models like CLIP [Radford et al., 2021] to convert textual inputs into latent vectors. These latent vectors associated with individual keywords represent specific control influences on images, which can be utilized for our quantitative evaluation task.

2.2 TIS Evaluation

The aesthetic score [Schuhmann et al., 2022] assesses the aesthetic quality of individual images, while the CLIP score [Radford et al., 2021] quantifies the similarity between generated images and prompts. Additionally, there exist metrics specifically trained to align with human preferences, such as [Wu et al., 2023b; He et al., 2022; Wu et al., 2023a; Xu et al., 2023; Kirstain et al., 2023; He et al., 2023a; He et al., 2023b]. Human preferences are intricate and encompass various dimensions, such as text-image similarity, image fidelity, aesthetics, and other factors. Among these metrics, HPSv2 stands out for its consistent scoring and use of larger, more diverse training datasets. Significantly, the CLIP model has been fine-tuned on an extensive dataset, featuring 798,090 human ranking choices across 433,760 image pairs [Wu et al., 2023a]. These evaluation metrics provide valuable visual feedback and enable a quantitative assessment of the multidimensional impact of keywords on TIS.

In this paper, we adopt a quantitative and multidimensional approach to evaluate the fused features of keywords. We use multiple TIS evaluation metrics, including HPSv2, the aesthetic score, and the CLIP score, with each metric representing distinct aspects of keyword effects.

2.3 Prompt Engineering for TIS

Due to the remarkable potential of TIS, there has been a surge of interest in prompt engineering. MagicPrompt, a widely adopted model for automatic prompt completion, is trained on high-quality prompts sourced from the Internet. BeautifulPrompt [Cao et al., 2023] utilizes a reinforcement learning technique, complemented by visual AI feedback, to finetune language models and maximize the rewards of generated prompts. However, these methods primarily focus on completing or rewriting prompts, which may inadvertently alter content entities, resulting in rigid and inflexible generated prompts. Liu et al. [Liu and Chilton, 2022] conducted a series of experiments, proposing several design guidelines for text-to-image prompt engineering that confirm the beneficial impact of incorporating keywords. Additionally, [Oppenlaender, 2023] identified six distinct types of keywords through an ethnographic study spanning three months within the online generative art community.

However, previous studies have been constrained by the laborious and time-consuming manual engineering of prompts. In contrast, BestPrompt [Pavlichenko and Ustalov, 2023] employs a genetic algorithm to identify keywords that compose prompts, aiming to achieve optimal aesthetic quality in images. Nevertheless, this study solely relies on selecting a fixed set of keywords based on average performance determined through human evaluation.

Conversely, AK4Prompts can assess the varying impact of keywords and select the most suitable ones for specific prompts, thereby enhancing the visual appeal of generated images according to user preferences. To address the potential variations in the effects of keywords across different prompt categories and provide personalized keyword evaluation based on prompt content, we decouple the latent representations of keywords and prompts in TIS models, which have distinct impacts on the generated images. Subsequently, to further enhance this integration, we employ multiple crossattention layers to merge the semantic features of prompts with those of keywords.

3 Data Collection for Training

Simple Prompt datasets. The image-caption datasets, such as COCO Captions [Chen et al., 2015] and LAION [Schuhmann et al., 2022] primarily describing real images, may not align with the interests of generative model users. DiffusionDB [Wang et al., 2022] comprises a large-scale dataset filled with a wide range of user-written prompts and generated images. However, it features numerous complex keyword-based text prompts. Regarding our prompt dataset, we require simple text prompts to maximize the impact of different keywords. SUR-adapter [Zhong et al., 2023] employs a pre-trained BLIP model to generate captions for images associated with high-quality prompts, treating the brief and less detailed resulting captions as low-quality prompts. BeautifulPrompt [Cao et al., 2023] enhances this approach by using ChatGPT [OpenAI, 2023] to summarize highquality prompts, treating these summaries as low-quality prompts. The dataset includes 143k simple prompts and 2k test prompts. We train our model using BeautifulPrompt's training set and evaluate its performance on the test set.

Keywords Collection. We extract a series of keywords from complex prompts in DiffusionDB, separating them by commas and sorting them based on frequency of occurrence. After manually removing non-keywords and those with lower frequencies, we obtain a list of 51,291 keywords. However, the keyword set contains numerous semantically consistent and repetitive terms. Since the SD utilizes CLIP to encode text inputs into latent vectors for conditional control, keywords with similar CLIP encodings have comparable effects on the generation outcome. To reduce keyword redundancy and prevent the model from developing a specific bias, we employ CLIP to uniformly encode the keywords, compute their similarity matrix, and apply the k-means clustering algorithm for grouping them into clusters. We retain the most frequently occurring keywords in each cluster while removing redundant ones to reduce the overall number of keywords further. Through extensive experimentation, we ultimately select 954 distinct semantic representations as our final keyword set. Our AK4Prompts can generalize to all 51,291 keywords via CLIP encoding. Keywords with similar semantics receive similar scores, even rare ones, matched based on learned semantic similarities.

4 AK4Prompts

As shown in Figure 2 (Top), let us consider the keywordprompt pairs (pi, ki)Ni=1 , where pi represents the original prompt and ki denotes a randomly selected keyword from the collected set of keywords. Firstly, we freeze all learnable parameters of the text encoder fEn and the predictor fpre in the pre-trained TIS model, as well as all the evaluation models used. Then, we utilize the TIS model to generate images Ii corresponding to keyword-based prompt ˆ pi and calculate average scores for both images Ii and the original prompt pi using various evaluation models.

Finally, we propose our trainable AK4Prompts model to predict these scores based on the features of pi and ki from the text encoder. Our model, as illustrated in Figure 2 (Bottom), comprises two components: the Semantic Fusion Module (Section 4.1) and the Multidimensional Score Prediction Module (Section 4.2). Section 4.3 demonstrates how to utilize AK4Prompts for inference by identifying suitable keywords according to user prompts and customized preferences.

4.1 Semantic Fusion of Keywords and Prompts

The structure of the Semantic Fusion Module is depicted in Figure 2 (Bottom), which comprises L cross-attention layers. These layers are responsible for fusing the semantic features of keywords and prompts, generating attention maps for each textual token of prompts. Each layer consists of four learnable transformations, denoted as hj(·) for j = 1, 2, 3, 4, implemented using a multi-layer perceptron (MLP). We utilize the outputs fEn(ki) and fEn(pi) from the text encoder as initial embeddings for keyword features embki 0 and prompt features embpi , respectively.

Then, we construct Qi = h3[embki ℓ ] and Ki = h2[embpi ], and calculate attention values as described in [Devlin et al., 2018; Vaswani et al., 2017]

where d represents the feature dimension of Qi and Ki, and the cell atti,j ℓ defines the weight of the value of the j-th token in pi on ki. Additionally, we construct Vi = h1[embpi ], and obtain the calibrated semantic information as ˆ Vi = Vi ⊗ atti, which is then used to update the keyword features embki . Afterwards, the output of the first cross-attention sublayer is transformed by a learnable transformation function denoted as h4(·) to obtain the fused semantic features.

Here, We employ a residual connection [He et al., 2016] around each of the two sub-layers, followed by layer normalization [Ba et al., 2016]. Finally, the output of the Semantic Fusion Module is obtained as embki L through the crossattention layers.

4.2 Multidimensional Score PredictionModule

The Score Prediction Module consists of three MLP score prediction heads, ˆ Shps(·), ˆ Sclip(·) and ˆ Saes(·), as shown in Figure 2 (Bottom). Each prediction head utilizes the output keyword embedding from the Semantic Fusion Module to predict the average HPSv2 score [Wu et al., 2023a], CLIP similarity score [Radford et al., 2021], and aesthetic score [Schuhmann et al., 2022] of the generated image based on the corresponding keyword prompt.

Briefly, to enhance the prompt pi with keyword ki, append ki to the end of pi and form ˆ pi. Then input ˆ pi into the downstream TIS model to generate image Ii:

HPSv2 is a human preference model trained on a large dataset of text-to-image prompts and real user preferences. To mitigate the impact of random seeds on the quality of images generated by the TIS model, we employ 8 different random seeds to generate images and average the results. The calculated averaged HPSv2 score HPS is used as the ground truth for training our model. The loss function is as follows:

where ˆ Shps(embki L ) represents the scalar output of the HPSv2 prediction head for the keyword embedding embki L obtained from the Semantic Fusion Module. MSE refers to Mean Squared Error, and N denotes the total number of samples.

CLIP score is a metric used to evaluate the correlation between the image generated by the model and the original text. In our work, we use a MLP prediction head to assess the consistency between the original prompt pi and generated image Ii under the influence of keyword ki. The calculated average CLIP score CLIP is used as the ground truth to train our model. The loss function is:

The aesthetic score model is trained to predict the rating that people give when asked "how much do you like this image on a scale from 1 to 10". To maintain consistency with HPS and CLIP, we scale the aesthetic scores to a range of 0 to 1. Similarly, a MLP prediction head is trained to fit the corresponding keywords to the aesthetic scores AES of the generated images:

Finally, we combine the losses of the three MLP prediction heads as the final Ltotal using loss coefficients h, a, and c.

4.3 Customized Keywords-Ranking

After training, our AK4Prompts gak(·) can predict scores for each keyword based on simple prompts, allowing us to determine the optimal set of keywords that align with these prompts. For a given simple prompt p, we input it into our model to simultaneously predict HPSv2 scores, CLIP scores, and aesthetic scores for all keywords in a parallel inference

where ki represents a keyword and K denotes the total number of keywords. Then, we can calculate a comprehensive score for each keyword ki as follows:

where (ˆh, ˆ c, ˆa) represent the weights for the HPS score, CLIP score, and aesthetic score, respectively.

Finally, a comprehensive ranking of all keywords can be obtained, where higher-ranked keywords exhibit a more pronounced enhancing effect on the prompt p. Moreover, by customizing the values of (ˆh, ˆ c, ˆa) based on user preferences, keywords with different emphases can be sorted and selected. For instance, assigning a larger value to ˆa enables the selection of keywords that exert a greater impact on aesthetics. Based on this keyword ranking, we can choose the topperforming keywords and append them to simple prompts in order to enhance overall image generation performance.

Our method achieves higher computational efficiency compared to prompt generation models due to its use of precomputed semantic embeddings, avoidance of self-attention computation during keyword processing, and smaller model size.

5 Experiments

Training Settings. Our model was trained on 143KBeautifulPrompt training samples. We utilized the pre-trained stable diffusion 1.5 as our TIS model and employed LCM-LoRA for accelerated inference. We generated 512x512 resolution images through a four-step inference with a CFG scale ω set to 1.0, leveraging FLOAT16 formats to save GPU memory and speed up training. For the semantic fusion module, we set L = 3. Regarding Ltotal, we set h = 2.25, c = 2.25, and a = 1. Due to different ranges in the distributions of these scores, we assigned a larger weight to the one with smaller ranges to balance gradients and enhance prediction accuracy. The learning rate was set at 1e-4, weight decay at 1e-2, batch size at 32, and training step at 88,000 steps. All experiments were implemented in PyTorch and run on a single server with NVIDIA RTX3090TI GPUs.

Baselines. We consider two strong baselines: BestPrompt [Pavlichenko and Ustalov, 2023] and MostPopular. BestPrompt is the first work to identify the best keywords for TIS, using a genetic algorithm that identifies a set of high-quality keywords through extensive human evaluations. MostPopular represents the most frequently used keywords in TIS, which have gained widespread recognition within the user community for their proven effectiveness in enhancing the quality of generated images. Our method requires setting the number of selected keywords and specific preferences. Whereas the generation methods may produce new prompts with different keyword lengths, making comparison challenging.

Evaluation Protocols. We assess the images generated by the TIS model using 2K BeautifulPrompt testing prompts employing various keyword selection methods. Multiple TIS models are employed to generate images, and we compute the aesthetic score [Schuhmann et al., 2022], HPSv2 [Wu et al., 2023a], and CLIP score [Radford et al., 2021] for both the images and original prompts. For the baselines, we evaluate using different numbers of keywords, specifically top-1, top5, and top-10. Regarding our AK4Prompts approach, we set (ˆh=3, ˆ c=5, ˆa=1) to determine a comprehensive ranking for all keywords and select an equivalent number of top keywords for evaluation and comparison purposes. As a comparative measure due to baselines' emphasis on clip scores, we also assign a higher weight to ˆ c in order to enhance the clip score.

Additionally, we set (ˆh, ˆ c, ˆa) as (1, 0, 0), (0, 1, 0), and (0, 0, 1) respectively to select the top-5 keywords that have a significant impact on the aesthetic score, CLIP similarity score, and human preference score for evaluation.

5.1 Overall Results

Our method consistently outperforms the other baselines in most scores, as demonstrated by Table 1. Given that the CLIP score reflects semantic consistency between text and image, it is unsurprising that sending the original prompts without any additional keywords to Stable Diffusion unchanged yields the highest score. Our approach achieves superior aesthetic scores, HPSv2, and CLIP scores compared to the baselines while maintaining a comparable CLIP score to the Original prompt. This highlights the effectiveness of our proposed method, AK4Prompts, in selecting optimal keywords for input prompts.

Our AK4Prompts is applicable to all SD-style models. The experimental results of SD1.5 and SDXL-Turbo demonstrate the transferability of AK4Prompts to other diffusion-based TIS models. Taking into consideration the widely adopted model SDXL-Turbo 2 , Figure 3 illustrates that, despite already performing well with most vanilla prompts, SDXLTurbo can further enhance its image generation capabilities when utilizing AK4Prompts. As depicted in Table 2, the selection of only the top 5 keywords, predicted by the model with the highest aesthetic scores for image generation, leads to a significant enhancement in aesthetic scores, as well as CLIP and HPS scores. This observation signifies that our approach proficiently captures diverse keyword effects, offering guidance to users and enabling customization of final output preferences. Additional examples are provided in Appendix A and B in our code repository, including qualitative analyses of different keywords and the specific effects of different categories of keywords on image generation. We also feature a qualitative comparison of images generated using keywords selected with different preferences.

2 https://huggingface.co/stabilityai/sdxl-turbo

5.2 Ablation Study

In Table 3, we analyze the various design choices implemented throughout our experiments. As discussed in Section 4.1, the Semantic Fusion Module integrates the semantics of simple prompts with keywords. This integration enables a more effective combination of different prompt semantics, thereby providing personalized keywords. To validate the module's effectiveness, we initially compare it with a simpler method that omits the Semantic Fusion Module. This method involves directly using the text encoder to obtain semantic encodings of user prompts and additional keywords, followed by score prediction using MLP prediction heads. Subsequently, we employ self-attention instead of cross-attention for integrating semantic features from user prompts and additional keywords. Finally, we evaluate different scheduling steps using LCM-LoRA. For a fair comparison, both models were trained for 88,000 steps under identical configurations.

6 Conclusion

In this paper, we propose a universal algorithmic framework named AK4Prompts that quantitatively assesses the different effects of various keywords in TIS models based on simple prompt. Our approach significantly reduces the complexity of prompt design and the difficulty of keyword selection, thereby saving users considerable trial-and-error time. While AK4Prompts can automatically select keywords for user prompts and generate more aesthetically pleasing images, we find it challenging to strike a balance among factors such as image aesthetics, human preferences, and text-image consistency in order to choose the best keywords. At times, in pursuit of better text-image consistency, we have to compromise on certain aspects of image aesthetics. Additionally, since we use SD1.5 and employ LCM-LoRA with CFG scale ω set to 1.0, only positive keywords are used to control generation while negative keywords are not considered. Similar to positive prompts, negative prompts are also crucial inputs for many TIS models. The final image generation results depend on both positive and negative prompts. Therefore, discovering the best negative keywords can effectively improve the quality of generated images as well. The incorporation of negative keywords into our framework is achievable, but addressing this aspect will be a key focus in the development of AK4Prompts 2.0.

Acknowledgements

This work was supported by the Innovation Research Group Project of NSFC (61921003).

Contributions

• † on Page 1 indicates that Haiyang Zhang, Mengchao Wang and Shuai He, three authors of this paper, have made equal contributions.