BATON: Aligning Text-to-Audio Model Using Human Preference Feedback

1 Introduction

Text-to-audio (TTA) generation is a promising application that concentrates on synthesizing diverse audio from text prompts. Recent advances in diffusion-based generative models, such as AudioLDM [Liu et al., 2023a], Make-AnAudio [Huang et al., 2023b], and TANGO [Ghosal et al., 2023], have significantly facilitated audio generation. These models employ latent diffusion models [Rombach et al., 2022] to create high-fidelity audio based on textual content, surpassing the performance of previous state-of-the-art TTA models. Nevertheless, there remains a bias between input text and generated audio, caused by the inherent information density of natural language and limited model ability. As shown in Figure 1, the upper example shows low integrity due to a missed audio event, and the lower example exhibits an incorrect temporal relationship because of a wrong time order.

Recent studies have made efforts to mitigate this bias. For instance, Make-An-Audio2 [Huang et al., 2023a] utilized large language model to generate structured captions, addressing issues of poor temporal consistency. ReAudioLDM [Yuan et al., 2024] tackled the missing generation of rare audio classes through retrieval augmentation. Auffusion [Xue et al., 2024] explored the capabilities of different text encoders to capture fine-grained information such as temporal order and fine-grained events. However, none of these approaches leverage human preference feedback, extensively applied in language [Ouyang et al., 2022; Achiam et al., 2023] and image generation [Lee et al., 2023], to address the aforementioned issue. Therefore, we aim to explore a new perspective: Can effective alignment between text prompts and generated audio be achieved through human preference feedback in text-to-audio generation?

To investigate this, we propose a new framework named BATON, designed for aligning TTA models using human feedback. To the best of our knowledge, BATON is the first work to improve the alignment of TTA models through human preference feedback. The BATON involves three crucial steps, as depicted in Figure 2: (1) We generate initial text prompts using GPT-4 [Achiam et al., 2023], conditioned on audio labels obtained from AudioCaps [Kim et al., 2019] and a self-build conjunction list, as it has proven to be a reliable annotator [Yang et al., 2023c]. Next, audio is generated for each text prompt using an advanced TTA model, resulting in text-audio pairs. Notably, the generated multi-event audio emphasize the integrity and temporal relationships, aspects that pose challenges for existing TTA models [Xue et al.,

2024; Liu et al., 2023a; Ghosal et al., 2023]. Then, we collect binary human feedback for these pairs. (2) We introduce and train an audio reward model, building upon the humanannotated dataset. This model is tailored to predict human feedback based on the provided textual input and corresponding audio. (3) We fine-tune an off-the-shelf TTA model, TANGO [Ghosal et al., 2023], through reward-weighted likelihood maximization to enhance the alignment of the TTA model with human preference, specifically focusing on the alignment between input text and generated audio. Here, the pre-trained loss of TANGO [Ghosal et al., 2023] is also incorporated as a regularization mechanism, preventing the model from overfitting to the annotated dataset.

We conducted extensive experiments to illustrate that BATON is capable of achieving significant improvements in text-audio alignment. Specifically, BATON demonstrates gains of +2.3% and +6.0% in CLAP scores for integrity and temporal relationship tasks, respectively. When evaluated by human annotators, BATON achieves a MOS-Q of 4.55 for the integrity task, surpassing the original model's score of 4.05, and attains a MOS-F of 4.41 for the temporal relationship task, outperforming the original model by 0.58 MOS-F.

In summary, our main contributions are as follows:

• We generate a dataset with 4.8K text-audio pairs across 200 audio event categories, in which 2.7K samples are annotated by human annotators.
• Using the constructed dataset, we train an audio reward model to predict the TTA alignment score reflecting human preference. Our experiments demonstrate that scores predicted by the audio reward model closely correlate with human preference.
• The audio reward models are utilized to enhance an offthe-shelf TTA model in terms of the integrity and temporal relationship of audio events. Detailed experiments provide strong support for its effectiveness.

2 RelatedWorks

2.1 Text-to-Audio Generative Models

The TTA generative models are designed to generate audio that is semantically consistent with text prompt. Currently, their predominant architecture is founded upon the latent diffusion model [Rombach et al., 2022; Ho et al., 2020]. These models engage in a process of noise injection and denoising within the latent space, which is obtained by encoding audio features. And the generated audio representation undergoes transformation into a waveform through a vocoder [Kong et al., 2020a].

Diffsound [Yang et al., 2023a] employs a discrete diffusion model for audio generation from text, operating within the latent space obtained by quantized VAE(VQ-VAE) [Van Den Oord et al., 2017] trained on mel-spectrograms. On the other hand, Audiogen [Kreuk et al., 2022] is built upon a VQ-VAE trained on raw waveform and utilizes an autoregressive language model for the generation of audio guided by text. For the continuous latent space, AudioLDM [Liu et al., 2023a] utilizes audio features obtained from contrastive language-audio pretraining (CLAP) [Wu et al., 2023] as training conditions and guides the generation of audio through textual features encoded by CLAP, which effectively compensates for the scarcity of paired data. TANGO [Ghosal et al., 2023] harnesses the remarkable text representation capability of the pretrained Large Language Model (LLM), achieving superior performance within the same latent space as AudioLDM, even with limited paired data.

However, challenges remain for generating audio that matches the details of prompt and human perception. Some recent studies have tried to handle this problem in terms of prompt processing and model components design. MakeAn-Audio2 [Huang et al., 2023a] employs LLM to generate structured captions to tackle poor temporal consistency. Yuan [Yuan et al., 2024] solves missing generation of rare audio classes with retrieval augmentation. Guo [Guo et al., 2024] uses timestamps condition for controllable audio generation. Auffusion [Xue et al., 2024] explores the ability of different text encoders for capturing fine-grained information like temporal order and fine-grained events.

2.2 Fine-Tuning with Human Preference Feedback

Fine-tuning guided by human preference feedback, stands as an indispensable component within the Reinforcement Learning from Human Feedback (RLHF). This approach proves particularly vital in machine learning, especially when dealing with intricate or ambiguous objectives. Its efficacy spans diverse applications, ranging from gaming, as demonstrated in Atari [Christiano et al., 2017], to more intricate tasks in robotics [Ziegler et al., 2019; Casper et al., 2023], where it significantly enhances the agent's success rate in completing tasks.

The integration of RLHF into the development of large language models (LLMs) signifies a noteworthy milestone in the field. Notable models such as OpenAI's GPT-4 [Achiam et al., 2023], Anthropic's Claude [Anthropic, 2023], Google's Bard [Google, 2023], and Meta's Llama 2-Chat [Touvron et al., 2023] leverage this approach to improve their performance and relevance. Collecting human judgments on response quality is often more feasible than obtaining expert demonstrations. Subsequent works have fine-tuned LLMs using datasets reflecting human preferences, resulting in enhanced proficiency in translation [Kreutzer et al., 2018], summarization [Stiennon et al., 2020], story-telling [Ziegler et al., 2019], and instruction-following [Ouyang et al., 2022]. Additionally, RL and RLHF has been applied to enhance spoken language understanding and train language models for various objectives [Ranzato et al., 2015; Wu and Hu, 2018; Cheng et al., 2023]. Presently, RLHF has been employed to fine-tune diffusion models, contributing to improved image equality, text-image alignment, and image aesthetic scores [Black et al., 2023; Dong et al., 2023; Yang et al., 2023b].

Similar to the RLHF method mentioned above, we also trained a reward model for evaluating the quality of audio and applied it to fine-tuning the pre-trained model. Differently, our approach does not use reinforcement learning algorithms to update the original model. Instead, we used the reward model as a weighting factor to increase the probability of audio with higher rewards.

3 Method

The framework of BATON, as shown in Fig 2, consists of three components. Firstly, we generated a synthetic dataset of audio-text pairs using selected prompts and then annotated it with task-specific human-derived annotations in Section 3.1. Furthermore, we employed the generated data, augmented with human feedback, to train a reward model designed to predict human preferences in audio content, which is illustrated detailedly in Section 3.2. Finally, as elaborated in Section 3.3, we utilized a reward model reflecting human preferences to fine-tune the text-to-audio model, thereby enhancing its task-specific alignment performance.

3.1 Dataset Construction

Recent studies [Huang et al., 2023a; Yuan et al., 2024; Xue et al., 2024] have found that text-to-audio models face challenges in producing audio aligning with human preferences, primarily manifest in low integrity, incorrect temporal relationships, and the like. As shown in Figure 1, the upper example exhibits poor integrity because of a missed audio event and the lower example has an incorrect temporal relationship due to a wrong time order. For simplicity, we here focus on evaluating the integrity and temporal relationships of generated audio to explore the effectiveness of human preference feedback in text-to-audio models. To start, we generate text-audio samples related to these two attributes, and subsequently, human evaluators rate these generated samples.

Data Collection. The left part of Figure 2 illustrates the process of generating audio data concerning integrity and temporal relationships. We specifically select audio event categories that ranked among the top 200 in occurrence within AudioCaps [Kim et al., 2019], constituting our meta labels Given a label group, Xlabel, the two labels in it will be described into an audio event group: {Event1, Event2}. Then, please join the audio event group with conjunction to form an audio caption, defined as "Event1, Conjunction, Event2".

Conjunction list: Xconj

Please ensure to randomly select conjunctions from the list and avoid relying on a single conjunction.

Please try to generate captions that match human language expressions as much as possible.

Table 1: GPT instruction for integrity task.

denoted as Xmeta = {l1, ..., l200}. Based on that, we randomly select 2 or 3 labels to create a label group, denoted as Xlabel. Conditioned upon the composed Xlabel and a predefined conjunction list Xconj, we instruct the GPT-4, named MT, with prompt P whereby attaining a complete sentence that matches human language:

The prompt P includes the prefix system message and the suffix prompt, guiding MT to produce the desired audio captions. The prompt for the group with 2 labels is detailed in Table 1. The conjunction list Xconj encompasses terms such as "and", "with", "followed by", and so forth (refer to the Appendix for the complete list). Following the generation process, we manually refine and rephrase certain captions to align them more closely with human oral language. As a result, Dtext encompasses a total of 962 audio captions, with samples presented in Table 2. Subsequently, we employ the TANGO [Ghosal et al., 2023], named MA, to synthesize 5 different audio for each audio caption:

Thus, we obtained 4, 810 text-audio pairs in total.

Human Annotation. Upon the collected dataset Ddata, we selected 2.7 K samples, denoted as Dhuman, and subjected them to evaluation by human annotators. As presented in Table 3, where 1, 431 samples are related to integrity, and 1, 332 samples are associated with temporal relationships. Annotators employed a binary rating system, assigning either 0 or 1 to each text-audio pair. This is because the nature of integrity or temporal relationship often yields a binary outcome (true or false). Annotators only need to make binary feedback instead of rating a continuous quality score. Admittedly, the more informative human feedback should be adopted for more complex or subjective aspects, such as audio quality and emotional expression. Nevertheless, binary scoring helps to minimize the effects of subjectivity and individual differences for the integrity and temporal relationship tasks, which improves the consistency of the assessment. In practice, we implement a task-specific human preference scoring system, where each audio sample receives an evaluation and is assigned a binary score reflecting its alignment with the given text. A score of 1 indicates adherence to the desired preference (aligned), whereas a score of 0 denotes non-conformance (not aligned). We have also set the option "skip" to help the annotators resolve ambiguities of the synthetic audio. As recorded in Table 3, 28.4% text-audio samples in the integrity task are incorrect and 59.4% text-audio samples in the temporal relationships task have wrong order, highlighting a substantial gap between generated audio and human preference.

3.2 Audio RewardModel

As shown in Figure 2, upon the established dataset Dhuman, we developed an audio reward model rϕ(c, x) (parameterized by ϕ) to predict a scalar reward, which reflects the human preference for the alignment between the provided text c and audio x. Such a reward model comprises a text encoder EC(·), an audio encoder EX(·), and MLP layers. When given the input text c and generated audio x, the text encoder and audio encoder extract the respective text embedding ec and audio embedding ex. Then, ec and ex are concatenated along the channel dimension and forwarded to the subsequent MLP layers. The last MLP layer yields a reward that signifies human preference. This process can be summarized as:

In practice, we exploit the audio encoder EX(·) and text encoder EC(·) from the CLAP model [Wu et al., 2023] that pre-trained on various text-audio samples. Since the human feedback in Dhuman is binarized, i.e., y ∈ {0, 1}, we consider the prediction of human preference as a classification problem. Accordingly, the audio reward model is trained by minimizing the binary cross-entropy loss:

The resulting audio reward model is capable of producing an alignment reward that emulates human preference.

3.3 Fine-Tuning the Text-to-AudioModel with Audio Reward

After obtaining the audio reward model rϕ(c, x), we combine it with the original conditional audio distribution p(x|c), resulting in a new probability distribution ˜ p(x|c) = f(rϕ(c, x))p(x|c). Here, f(·) is a monotonically increasing function and we choose f(·) = exp(·) by default. As the original generative text-to-audio model [Ghosal et al., 2023], our objective is to update the model p with parameters θ by maximizing the probability, i.e., minimizing the following loss function:

Since diffusion models [Ho et al., 2020; Rombach et al., 2022] are in principle able to model conditional distributions with a conditional denoising autoencoder ϵθ(xt, t, c), the Eq. 5 can be simplified as:

where xt is the noise version of the input x. Compared to the original denoising loss, the reward term behaves as a modulating factor: when an input sample aligns with human preference, the modulating factor is large, thus the text-toaudio model pays more attention to learning this good sample, and vice versa. In addition, since our constructed dataset is relatively small compared to the pre-trained dataset, we introduce the denoising loss from the original text-to-audio model [Ghosal et al., 2023]. Consequently, the final loss is:

in which Dpretrain refers to the sampled data from the pretrained dataset [Kim et al., 2019] and β is a hyper-parameter. This regularized loss prevents the model from overfitting to our self-build dataset, ensuring it does not deviate heavily from the original model.

4 Experiments

4.1 Implementation Details

Employing TANGO (Full-FT-Audiocaps) [Ghosal et al., 2023] as our baseline, we integrated a human feedback-driven fine-tuning framework. From the Audiocaps dataset, top occurrences 200 labels were chosen, and GPT-4 [Achiam et al., 2023] was utilized to expand these into prompts for textto-audio synthesis, resulting in a dataset of 962 audio captions and their 4810 resultant audio outputs of which default length is set to 10 seconds. Human annotators then binary score the text-audio pairs, assessing their alignment with preference criteria. We trained the audio reward model on synthetic dataset over 50 epochs, with a batch size 64 and learning rate 0.01 using the Adam [Kingma and Ba, 2014]. During the fine-tuning of the original model, we assigned a weight parameter β of 0.5 to the pretrain loss. The ratio of human-labeled data to reward model-scored data was maintained at 1:1 for both sub-tasks. Ddata and Dpretrain involve 4.8 K and 2.5K samples, respectively. The fine-tuning process, conducted over 10 epochs with settings including a learning rate of 1 × 10-5 , batch size of 6, and the default optimizer AdamW [Loshchilov and Hutter, 2017], utilized a total of 48GB*2 GPU memory.

4.2 Main Result

Evaluation method. For evaluation, we filtered the Audiocaps test set using words such as and, as, then, while, before, after, followed to obtain a multi-labeled test set. Subsequently, we curated subsets comprising 148 two-label and 165 three-label prompts from the filtered data. These two subsets are used to evaluate audio integrity and temporal consistency respectively. Consistent with prior research [Huang et al., 2023a; Liu et al., 2023a; Liu et al., 2023b; Xue et al., 2024], our objective evaluation of audio quality and fidelity employs metrics such as Fr´echet Distance (FD), Fr´echet Audio Distance (FAD) [Kilgour et al., 2019], Kullback-Leibler divergence (KL) [Yang et al., 2023a]. We adapt Inception Score (IS) to assess the quality and variety of samples. Additionally, we utilized the Cross-Modal Language-Audio Perceptual (CLAP) metric SCLAP to objectively assess the alignment between audio and text. For subjective analysis, 40 participants were recruited to rate the perceived quality and text alignment of the audio separately on a 5-point scale, with 5 being the highest possible score. The Mean Opinion Score (MOS) method, with a 96% confidence interval, is employed to evaluate two aspects: audio quality (MOS-Q) and the faithfulness of text-to-audio alignment (MOS-F).

Quantitative experiment. BATON showcases competitive performance across multiple metrics in text-to-audio tasks, as detailed in Table 4. In evaluating the similarity between generated and real samples, the FAD metric exhibits a slight reduction compared to the original model in both tasks. Considering that FD relies on PANNs [Kong et al., 2020b] classifiers and the labels in our fine-tuned dataset constitute a subset of the PANNs sound class, using FD as the primary metric for assessing audio quality appears to be more reliable. In temporal task, BATON surpasses the original model in terms of FD, IS and KL, indicative of its robustness in maintaining semantic cohesion while minimizing divergence from the target distribution. Additionally, BATON achieves the highest IS in both subtasks, showcasing excellent audio variety. Notably, the superior performance of BATON in SCLAP, particularly a 2.3% improvement over the baseline TANGO in the integrity task and a significant 6.0% enhancement in the temporal task, underscores its effectiveness in producing audio that aligns closely with human preferences. This enhancement can be attributed to our fine-tuning approach that leverages human

Original model

Finetuned model

feedback, allowing BATON to better understand and distinguish the nuances of audio content that are valued by human.

When subjectively evaluated on the integrity task, BATON demonstrates a superior MOS-Q of 4.55±0.05, outperforming TANGO (4.05±0.07). Concurrently, BATON's MOS-F is 4.42±0.05, indicating a more faithful text-to-audio alignment compared to TANGO's 3.70±0.07. Significantly, BATON continues to demonstrate superiority in subjective metrics for the temporal task, with its MOS-F surpassing the original model by 0.58 points. This observation indicates its adept handling of temporal information. These findings affirm that BATON not only generates high-quality audio that maintains or even exceeds the original model, but also sustains an audio reconstruction of the textual details.

Qualitative experiment. Figure 3 compares the original and fine-tuned models in terms of generation faithfulness for two-label integrity and three-label temporal samples. In our qualitative analysis, discernible discrepancies are noted in the audio generation by the original model. For integrity task, in response to the prompt "A motor hums softly followed by spraying", the generated audio included only one of the elements, but omitting specific audio event. Similarly, for temporal task, the original model misrepresented the sequence of events. An example is the prompt "A man talking as music is playing followed by a frog croaking", where the generated audio incorrectly prioritized the "frog croaking" event over the initial "man talking". In contrast, BATON demonstrates enhanced performance, achieving more faithful generation on diverse audio events and temporal order shown in prompts.

4.3 Ablation Study

Considering the increased intricacy posed by the three-label temporal task, which involves handling more audio events and challenges in temporal alignment, our ablation examination focuses on temporal task.

Human feedback. As detailed in Table 5, to dissect whether the enhancement in model performance is attributed to human feedback or increased data volume, we conducted a comparative analysis of the impact of data with and without feedback annotations on the model's performance. In the case without feedback annotations, 'PD', 'HD‡ ' and 'RD‡ ' denote fine-tuning with text-audio pairs data based on pretrain loss only. In the case with feedback annotations, we opted for BATON (HF). The baseline model TANGO achieved CLAP score of 35.9%. Incorporating 'PD' yielded slight gains in KL and CLAP accuracy but FD and FAD decreased, while continually integrating synthetic dataset 'HD‡ ' and 'RD‡ ' slightly improved CLAP score to 36.2% and 37.5%. Finetuning with human feedback achieves the highest CLAP accuracy at 41.9%, demonstrating this uptick is primarily attributed to the feedback of human preference instead of the boost of training data volume.

Preference data. The ablation study detailed in Table 6 investigates the influence of different preference data sources on BATON. Initially, employing only human-annotated data provides a CLAP score of 38.8%, indicating a moderate enhancement over the baseline. The exclusive use of reward model labeled data achieves an improved CLAP accuracy of 38.8%, suggesting that reward model alone is beneficial as well. The most substantial improvements were observed when both human-annotated and reward model labeled data are utilized together, optimizing the performance with an FD score of 36.19 and a peak CLAP score of 41.9%. This configuration demonstrates the synergistic effect of combining human annotation with reward score, significantly improving the generation with human preference.

Pretrain loss. The ablation study, outlined in Table 7 evaluates the efficacy of pretrain loss in fine-tuning. Incorporating pretrain loss improved the FD and FAD score to 36.2 and 4.31 while IS and KL score slightly decrease, and CLAP score rises to 41.9%, underscoring its significance in enhancing alignment and preventing overfitting on a relatively small amount of preference data.

Reward model. As shown in Table 8, comparison of encoders and training losses for the audio reward model reveals that the CLAP text encoder, coupled with BCE loss, significantly outperforms the Flan-T5 [Shen et al., 2023] encoder in test set performance, yielding superior alignment outcomes. This efficacy likely stems from ability of CLAP, through large-scale pretraining based on contrastive learning, to extract more aligned text feature and audio feature, and from aptness of BCE loss for binary classification (0 and 1) tasks within reward model training.

To assess the accuracy of reward model in predicting human preferences, we applied it to two subtasks test sets, each comprising 120 text-audio pairs, and annotated by two separate individuals. Figure 4 presents the findings. The predictions prominently feature two high-frequency peaks at 0 ("Not Aligned") and 1 ("Aligned"), suggesting a strong alignment of the model with human ratings. The concentration of predictions at these extremes, with few near the mid-point of 0.5, indicates the model's effectiveness in differentiating text-audio pair quality, particularly in tasks of integrity and temporal relationships.

5 Conclusion and Discussion

Conclusion. In this paper, we propose a novel framework (BATON), using human feedback to enhance the alignment between text prompts and generated audio in TTA models. Our BATON involves three steps: collecting human feedback for the constructed text-audio pairs, training an audio reward model on human-annotated data, and fine-tuning an off-theshelf TTA model using the audio reward model. Extensive experiments manifest that BATON effectively leverages human feedback to mitigate biases and improve alignment in TTA models, making a valuable contribution to the ongoing advancements in audio synthesis from textual inputs. We anticipate that BATON can pave the way for aligning TTA models through human feedback.

Discussion. As a plug-and-play fine-tuning framework for various TTA models, BATON has some limitations and areas for future exploration. Firstly, the data-driven characteristic of BATON means the performance of alignment depends on the quality and quantity of human feedback. Secondly, the reliance of our approach on a two-stage training process, rather than using reinforcement learning strategies for online finetuning, restricting BATON to offline updates. Future work should further explore reinforcement learning with human feedback in text-audio alignment.

Contribution Statement

Huan Liao and Haonan Han contributed equally to the design and implementation of the project, Jiasheng Lu and Xiu Li served equally as corresponding authors, providing supervision and guidance throughout the research. All authors have contributed to the design of the methodology, implementation of the experiments, evaluation of the model performance, and writing of the paper.