XAI-Lyricist: Improving the Singability of AI-Generated Lyrics with Prosody Explanations

1 Introduction

Composing singable lyrics for melodies has long played a central role in music composition. It weaves stories and emotions into melodies that resonate with listeners and singers [Barradas and Sakka, 2022; Brattico et al., 2011; Wang et al., 2022], enhancing the expressiveness of music and adding layers of depth to the auditory experience. To make lyrics composition faster and easier, the notion of automatic lyrics generation is emerging, mainly via Language Models (LMs) [Ma et al., 2021; Malmi et al., 2016; Watanabe et al., 2014; Potash et al., 2015; Sheng et al., 2021]. Despite their impressive performance in lyrics generation, most LMs only output plain text lyrics, making it difficult for people, particularly composers and singers, to understand whether the generated lyrics are suitable for singing melodies.

To address this limitation, we present XAI-Lyricist, leveraging musical prosody to guide language models in generating singable lyrics and providing prosody-based singability explanations. In the context of lyrics writing, musical prosody [Palmer and Kelly, 1992; Palmer and Hutchins, 2006; Heffner and Slevc, 2015; Everhardt et al., 2022] generally refers to the alignment of strong and weak beat notes, long and short notes in melodies, with stressed and unstressed syllables, long and short vowels in lyrics, respectively. This alignment harmoniously integrates linguistic and musical elements, resulting in lyrics that are rhythmically adaptable to melodic developments. In linguistic studies, musical prosody is considered fundamentally decisive for the singability of lyrics, viz., whether a piece of lyrics is suitable for singing [Khoshsaligheh and Ameri, 2016; G¨uven, 2019; Franzon, 2008]. Psychological studies have further shown the importance of musical prosody in helping people better sing and comprehend lyrical content [Gordon et al., 2011; Sahasrabuddhe, 2023], enhancing their linguistic and music skills [Patel and Iversen, 2007; Jansen et al., 2023], and helping children develop speech abilities [Caccia and Lorusso, 2021]. These findings support the need and reasonableness to communicate musical prosody as singability explanations for LM-generated lyrics.

Inspired by these insights, we argue that a singable piece of song lyrics should share a similar prosodic structure with its corresponding melodies (or expert-composed lyrics). To implement this, we use an encoder-decoder Transformer [Vaswani et al., 2017; Lewis et al., 2020] as the foundation model of XAI-Lyricist. We extend the Transformer as a multi-target language model, which encodes a prosody template as conditional input, and generates multiple target sequences: (1) lyrics; (2) the number of syllables in each lyric word; (3) the strength and (4) length of each lyric word. Besides presenting the output lyrics to people, XAI-Lyricist also generates demonstrations of lyrics' prosody patterns as singability explanations. A demonstration visualises the correspondence between melody notes and lyric words, rendering notes in different colours, and words in lower and upper case based on musical prosody. Figure 1 displays an example demonstration based on "Hey Jude" by The Beatles.

We used objective metrics and a human-subjects study to evaluate XAI-Lyricist, focusing on the singability of generated lyrics, and the usefulness of explanations. For objective metrics, we employed perplexity [Jelinek et al., 2005] and BLEU [Papineni et al., 2002], two established benchmarks in natural language processing [Chen et al., 2008]. We adjusted BLEU to prosody-BLEU as an indicator of singability, which compares the similarity between melody rhythm and lyrics' prosody (detailed in subsection 4.4). For the humansubjects study, we invited 14 participants with musical expertise. They were required to try singing melody sentences with given lyrics and rate the singability of each lyric sentence. Besides their subjective ratings, we also recorded the average time and number of attempts participants needed to sing each lyric sample, as objective measurements of singability.

The results show that incorporating musical prosody in a language model can improve its performance in singable lyrics generation, with significant improvements in the metrics mentioned above. Through the human-subjects study, we also demonstrate that musical prosody can faithfully explain the singability of LM-generated lyrics and is considered useful by participants. Moreover, subjective responses from the post-study interview suggest that communicating prosodybased explanations can help singers understand and judge the singability of lyrics faster, even enabling some of them to recognise unsingable lyrics and edit them to be singable. In summary, our main contributions include:

1. XAI-Lyricist framework for singable lyrics generation.
2. The provision of musical prosody-based explanations to help people to interpret the singability of song lyrics.
3. The evaluation of lyrics' singability and the usefulness of prosody explanations.

2 RelatedWork

This study uses musical prosody to improve and explain the singability of LM-generated song lyrics. We thus investigated previous studies and categorised them as follows:

2.1 Musical Prosody and Singability

In the context of lyrics writing, musical prosody generally refers to the alignment between lyrics' prosody and melodies' rhythm [Palmer and Hutchins, 2006]. To ensure a suitable lyric setting, human lyricists place stressed/unstressed syllables at strong/weak beats in melodies; long/short vowels at notes with long/short duration. The importance of musical prosody for singability has been discussed in psychological and linguistic studies. [G¨uven, 2019] proposes a hierarchical linguistic model of lyrical singability, where prosodic constraints have the most significant impact on singability among other layers. [Khoshsaligheh and Ameri, 2016] conclude that the prosodic match between lyrics and melodies is indispensable for producing a singable version of a song. [Franzon, 2008] suggests that the prosodic and poetic match suffices to make lyrics appear singable. Similarly, [Sogunro, 2022] states that the absence of musical prosody would make it technically impossible to sing lyrics. These findings suggest that the lack of musical prosody in automatic lyrics generation can be a research gap.

2.2 Automatic Lyrics Generation

Recent advances in AI have progressed automatic lyrics generation with generative models, such as recurrent neural networks (RNNs) for next-word prediction [Potash et al., 2015; Wu et al., 2019], Seq-GANs [Watanabe et al., 2018; Chen and Lerch, 2020], and Transformer-based models [Zhang et al., 2020; Tian et al., 2023; Sheng et al., 2021]. Moreover, some studies apply domain knowledge to LMs, providing finer-grained controls over the content of lyrics [Chang et al., 2021], such as syllable counts, keywords [Ma et al., 2021], and syllable lengths [Tian et al., 2023]. To the best of our knowledge, most existing LMs only output plain text lyrics without singability explanations, nor have they explicitly used musical prosody to control or explain lyrical singability. These limitations make it difficult for singers to properly align words with notes and for composers to assess if generated lyrics are singable. Therefore, we argue for the need to communicate musical prosody as singability explanations in automatic lyrics generation tasks.

2.3 Explainable AI inMusic Computing

Explainable artificial intelligence (XAI) aims to make AI understandable so that it can be useful to non-AI experts. In music computing, XAI can potentially advance music generative models by helping people understand music AI [Yan et al., 2023]. [Bryan-Kinns et al., 2023] explain the latent space of a variational autoencoder-based music generator. This work constrains four dimensions of the space to map four musical attributes and visualising their values. While the visualisation helps people comprehend the internal workings of a generative model, the four proposed features may be incomplete to fully explain music generation due to the intricate nature of music. Furthermore, [Zhao et al., 2019] use an additional LM to generate the reason behind each music recommendation in Users a human-like tone. However, LM-generated reasons can be less grounded predictions than faithful explanations. Another study [Wang et al., 2020] uses two music features, chord and texture, to control the generated music. This system generates a song from a given chord progression, enabling users to interpret the song by examining its chords. However, such music theory-based explanations are often complex and can overwhelm people. Furthermore, the intersection between explainable AI and automatic lyrics generation remains understudied. We are thus motivated to use musical prosody, a clearly defined concept in music psychology and linguistics, to develop a more explainable lyrics generation system.

Figure 2: The lyrics generation pipeline of XAI-Lyricist. Overall, the pipeline uses an encoder-decoder Transformer model to generate lyrics from a given prosody template (in the middle). During training (top left corner), the input template comes from the prosody (including syllable length and syllable strength) and metadata extracted from the lyrics corpus. To make the model better aware of musical prosody, we optimise it on four targets (right corner): lyrics, syllable numbers, syllable strengths, and syllable lengths. During inference (bottom left corner), the template comes from the rhythm of a melody and customised metadata. Specifically, syllable length and syllable strength are replaced by note length and note strength, respectively. The model then generates lyrics aligned with the melody rhythm constraints.

3 Technical Approach

3.1 SystemOverview

Figure 2 shows the lyrics generation pipeline of XAI-Lyricist. Overall, this pipeline uses an encoder-decoder Transformer [Vaswani et al., 2017], which generates lyrics from prosody templates. Given the lack of paired melody-lyric data available with accurate rhythm information, we resorted to a lyrics corpus for training, using the lengths (long/short) and strengths (strong/weak) of syllables in lyric words to create prosody templates (detailed in subsection 3.3).

To make the model better aware of musical prosody, we optimised the model on three prosody-related targets in addition to lyrics: (1) the number of syllables in each lyric word; (2) syllable strength (if the word has stressed syllables); (3) syllable length (if the word has long syllables). During inference, the model can take the rhythm of a melody to create the input prosody template, with syllable lengths and syllable strengths replaced by note lengths and note strengths, respectively. The model then generates lyrics whose prosodic structure aligns with the melody's rhythmic pattern, creating musical prosody. We also use the prosody of generated lyrics as singability explanations to people.

3.2 Musical Prosody

Melody Rhythm and Lyric Prosody

In music theory, beat refers to the basic unit of time. However, not all beats are equally important: some beats are considered stronger (strong beats), while others are weaker (weak beats). The rhythm of music is represented by a repeating sequence of strong/weak beats and long/short notes, analogous to lyrics whose prosody mainly comes from stressed/weak and long/short syllables. Musical prosody associates the rhythm of melodies with the prosody of lyrics. In this paper, we refer to the term prosodic lyrics as lyrics whose prosody is identical to the rhythm of their accompanying melodies.

Taxonomy of Notes and Syllables

We consider two attributes shared by melody notes and lyric syllables, strength and length, and categorise them as follows:

• Strong-beat notes and weak-beat notes refer to melody notes beginning at strong and weak beats, respectively. In 4/4 time music, the first and third beats are stronger than the second and fourth beats.
• Long notes and short notes: long notes are notes with relatively longer duration. Based on [Nichols et al., 2009], we define long notes as those whose duration is greater than the average duration of all notes in a melody phrase, with all non-long notes being short notes.
• Stressed syllables and weak syllables: stressed syllables are accented when pronounced and are marked by special symbols (["], [­]) in the international phonetic alphabet (IPA) 1 ; Weak syllables (a.k.a. unstressed syllables) do not have special markers in their IPA notations.
• Long syllables and short syllables: Long syllables have long vowels (marked by [:] in IPA) or diphthongs. A diphthong is a syllable with two vowel sounds, for example, sad ["sæd], hey [heI], etc. Syllables without long vowels or diphthongs are short syllables.

Taxonomy of Musical Prosody

Based on [Nichols et al., 2009] and our proposed taxonomy of notes and syllables, we define the following two types of musical prosody that make lyrics singable to melodies.

• Strength Alignment: Strong/weak-beat notes should concur with stressed/weak syllables, respectively.
• Length Alignment: Long/short notes should concur with long/short syllables, respectively.

1 https://www.internationalphoneticassociation.org/content/ full-ipa-chart

3.3 Data Representation

Prosody Representation

We use strength and length symbols to represent the rhythm of melodies and the prosody of lyrics. Melody notes and lyric syllables with similar attributes are notated by the same symbols, as shown in Table 1. For example, a strong-beat note and a stressed syllable are both notated by a <strong>. Similarly, a short note and a short syllable are marked by a <short>, by analogy.

For each melody note, we refer to its starting beat and duration to obtain its strength and length attributes. For each syllable in lyrics, we use a Python package named Prosodic 2 to query its IPA symbol and obtain its strength and length attributes (more details are included in section 1 of the supplementary document 3 ). To reduce the length of prosody representations, we compress each pair of strength and length symbols into one compound word, following [Hsiao et al., 2021]. For example, a strong-beat long note (or a stressed long syllable) is notated by a <strong,long>; a weak-beat short note (or a weak short syllable) is a <weak,short>.

Prosody Templates

A prosody template for a song includes the title and metadata of the song, followed by the prosody representation of each sentence in this song. Specifically, each template starts with a <title>, followed by the song name. Then, a <sent x> is used to mark the beginning of a sentence, with x being the order of the sentence. After each <sent x>, the keywords and prosody representation of the sentence are appended, prefixed by a <keywords> and a <prosody>, respectively. An example template is shown in the middle of Figure 2. Before being input into the Transformer model, each symbol in the template is tokenised and embedded as a vector. For compound words, the strength and length symbols are embedded as two vectors, which are then concatenated and linearly projected to the embedding size of the model, as in [Liang and Wang, 2024; Zhang et al., 2023].

Lyrics and Other Target Sequences

The decoder end generates four target sequences, each using a different vocabulary. The lyrics target uses an English word vocabulary with punctuation marks and special beginning (<BOS>) and end of sequence (<EOS>) symbols. The syllable strength and length sequences adopt symbols in Table 1 as their vocabulary. For the syllable number target, we count the maximum number of syllables per sentence in our lyrics dataset and initiate a new vocabulary with 20 symbols formatted as <syllable s>, where s ranges from 1 to 20. The syllable number, strength and length vocabularies also include an additional <pad> symbol to accommodate special non-word symbols in the lyrics target, e.g. timestep 1 in Figure 2. At each input timestep, the embeddings of four symbols from four target sequences are concatenated and linearly projected to the model embedding size.

2 https://pypi.org/project/prosodic/

3 Codes and supplementary materials are available at: https: //github.com/lqhac/XAI-Lyricist

3.4 Generating Lyrics and Explanations

Sampling Lyric Words and Prosody Symbols

The decoding uses top-k temperature sampling. At each decoding step, we first sample a symbol from the lyrics vocabulary. If the sampled symbol is a word, then the other three prosody-related symbols of this word are sampled. If the sampled symbol is not a word (e.g. a comma), the sampling of the other three symbols is skipped. The decoding ends once an <EOS> is sampled from the lyrics vocabulary.

Prosody Correction

Since the prosody symbols inferred by LMs may not be completely accurate, we also refer to the IPA of each sampled lyric word for prosody correction. Namely, inferred prosody should be replaced by standard IPA prosody if they conflict.

Demonstrations of Singability Explanations

We visualise the melody, generated lyrics, and strength/length symbols as a demonstration of singability explanations. An example is shown in Figure 1. First, we align all lyric words with melody notes using the two types of musical prosody in section 3.2. Then, all melody notes are rendered as rectangles on a piano roll, with their colours/widths indicating their strengths/lengths, respectively. For lyric words, all stressed syllables are capitalised and annotated on their aligned notes.

4 Experimental Setup

The following experiments aim to address two research questions (RQs) about musical prosody:

RQ1: Can musical prosody guide a language model to generate more singable lyrics?

RQ2: Is it useful to communicate musical prosody as singability explanations to people?

To this end, we implemented three LM-based lyrics generation models with/without prosody constraints and explanations. Using the lyrical content generated by these models, we conducted objective evaluations and a human-subjects study. This section describes the dataset, baselines, model configurations, metrics, and the setup of the human-subjects study. The results are reported in section 5.

4.1 Models

We implemented the following three models in the experiment. To allow for direct comparisons, we adopted the same lyrics vocabulary sfor all models.

• Vanilla, which directly generates lyrics from melodies without prosody constraints. We followed the methodology of [Sheng et al., 2021] to implement this baseline.
• Prosody, which generates only prosodic lyrics but does not provide singability explanations.

• Explainable Prosody (XAI-Lyricist), which generates both prosodic lyrics and singability explanations.

Both Vanilla and Prosody generate only plain-text lyrics, while Explainable Prosody additionally presents demonstrations of singability explanations as in Figure 1.

4.2 Datasets

For Vanilla, we followed [Sheng et al., 2021] and used their datasets for training and validation. For Prosody and Explainable Prosody, we used pop English song lyrics collected from a genre classification dataset 4 for training and validation. We excluded all non-English samples and used the Prosodic package to query the IPA of each lyric sample to obtain prosody templates. All samples with out-of-vocabulary words were filtered. The final dataset after cleansing includes 101,120 full song lyrics. For the test data in the human-subjects study, we randomly collected 210 sentences with low popularity on NetEase Music 5 as the test database. The selected sentences were manually transcribed into MIDI melodies. It is confirmed that none of the participants knew the selected songs before the human-subjects study.

4.3 Model Configuration and Training

All models employ a 6-layer encoder-decoder Transformer [Vaswani et al., 2017; Lewis et al., 2020], with 8 attention heads and a hidden size of 768. The feed-forward channel is 2048; the dropout rate is 0.3. The model was trained on three NVIDIA RTX A5000 GPUs. For the learning rate, we used Adam optimiser with β1 = 0.9, β2 = 0.98, ϵ = 10-5 . For the loss function, Vanilla and Prosody were optimised only on the lyrics' cross-entropy loss, while Explainable Prosody (XAI-Lyricist) employed the summation of crossentropy (CE) losses for four target sequences:

4.4 Metrics

• Perplexity: Perplexity is a benchmark to assess language models [Jelinek et al., 2005]. A lower perplexity can indicate higher fluency of LM-generated sentences.
• Prosody-BLEU: We adjust BLEU [Papineni et al., 2002] to prosody-BLEU as an indicator of singability. It quantitatively measures the similarity between lyrics' prosody and melody rhythm under the prosody representation shown in Table 1.
• Human Rating reflects the degree to which a participant subjectively feels a lyric sentence is suitable to be sung with a melody, on a 5-point scale.
• Average Time (T) refers to the time taken by a participant to successfully align and sing a lyric sentence with its melody, normalised by the length of the lyric sentence. We suppose that, averagely, participants tend to take longer/shorter time to align less/more singable lyrics to melodies. T can be formulated as:

4 https://www.kaggle.com/datasets/mateibejan/

multilingual-lyrics-for-genre-classification

5 https://music.163.com/

where L denotes a lyric sentence; t denotes the actual time taken to align L to its accompanying melody; |L| is the number of syllables in L. The logarithm scales the magnitude for better data analyses and visualisation.

• Average Attempt Number (A) is the number of attempts a participant needs to successfully align a lyric sentence to melodies, normalised by the length of the lyric sentence. An attempt refers to a one-time trial to sing out the lyrics that a singer aligns with the melody in a subjectively natural and fluent feeling. For example, if a participant finishes singing at one time without pausing or repeating the same parts, the attempt number is 1. If a participant pauses and repeats an identical part during singing once (e.g. "Find my... Find my way ..."), the attempt number is incremented by 1. A is formulated as:

where L denotes a lyric sentence; natt denotes the actual number of attempts to align L with its accompanying melody; |L| is the number of syllables in L.

4.5 Human-subjects Study

Participants

We recruited 14 participants, each with at least 7 years of experience in formal music training (with degrees or certifications in music and lyrics composition, performance, singing, etc.). All participants have prior experience in singing English songs. Each participant received compensation equivalent to about US$14.89 after their successful completion of the study. Breaks were permitted during the study to counterbalance potential fatigue-related side effects.

Apparatus

Each participant was presented with 99 shuffled sentences randomly selected from the test database. We used sentencelevel experimental design because: (1) it reduces the workload and does not overwhelm participants in each singing trial; (2) there are often repetitions between consecutive sentences in a song [Lloyd, 2020]; (3) with multiple sentences, participants tend to infer next sentences during singing [Hansen et al., 2021], which can affect the study. The 99 samples were generated by the three models, each model taking up 1/3. The sentence length of each model ranges from 2 to 11 syllables (mean=6.17; median=6; SD=2.09). The lyrics and melody for each sentence were displayed on a digital audio workstation for playback and visualisation. Figure 1 is an example demonstration of Explainable Prosody. For the other two models without explanations, the visualisations only included notes on the piano roll, and participants read plain text lyrics annotated beside the piano roll (more examples are shown in section 2 of the supplementary). The duration of the experiment ranged from 45 to 90 minutes.

Procedure

1. Participant Consent Participants first read the introduction and sign the consent form with IRB approval. The study was audio recorded after participants' consent.

1. Pre-study Tutorial We offered a pre-study tutorial with 12 lyric samples to help participants understand the task. The 12 samples are equally generated by three baselines, but excluded from the 99 sentences for the formal study. Participants were instructed to use the samples to sing the melodies. Then, they needed to independently and correctly complete 6 new test samples meeting the prevalidated standard alignment, before the formal study.
2. Melody Familiarisation Since participants' unfamiliarity with melodies can affect their singing and judgement of singability, participants were required to first familiarise themselves with the melody by listening to its piano rendition before reading the lyrical content. They were allowed to play the rendition an unlimited number of times, until they could hum the melody at least three times without pauses in between or replaying it.
3. Singing Trial After the familiarisation, participants read the lyrics (and explanations). They needed to try to align the lyrics with the melody and sing the lyrics out. Participants were explicitly told not to care about how well they could sing. Namely, they needed not take more trials to improve singing techniques, singing voice quality, music intonation, etc. They could stop once they felt they had already made a satisfactory lyric-melody alignment, before which unlimited attempts were allowed.
4. Singability Rating After singing of each lyric sample, participants rated the singability of the sample on a 5point scale. They were told to consider only how suitable they felt the lyrics were to be sung with the melody. Steps 3-5 were repeated for all 99 samples before step 6.
5. Post-study Briefing We finally conducted a post-study briefing for participants, clarifying the study's objectives and necessary details. Participants were encouraged to ask questions and share their thoughts about the study.

5 Results

5.1 RQ1: CanMusical Prosody Guide a Language

Model To Generate More Singable Lyrics?

Musical Prosody Improves the Singability of LM-Generated Lyrics

From Table 2, we can observe that compared to the Vanilla baseline, Prosody and Explainable Singability have lower perplexity, higher prosody-BLEU and human singability ratings. These indicate that infusing musical prosody helps LMs generate more fluent and singable lyrics for melodies.

Similarly, during the human-subjects study, all participants said that they encountered some lyric samples "incoherent" (P1, P3), "strange" (P2, P4, P6), "did not make sense at all" (P7). We observed that these problematic samples all came from the Vanilla model, with grammar issues (e.g. "Stains the planets in the sea"), odd coinages (e.g. "burn the tears I arteaths"), word repetitions (e.g. "I fight fight forever. Chance I I could live"), etc.

We suppose that these problems can be attributed to the "vagueness" of Vanilla. The Vanilla model uses a sequence of pitch-duration pairs (e.g. <G3><1/16>) to represent a melody and directly generates lyrics from this melody representation. However, the connection between pitch-duration pairs and lyric words is often vague, as each pitch and duration can irregularly correspond to many different words. Therefore, it can be confusing for LMs to converge well on such melody-lyric data, resulting in high perplexity, less singable, and even problematic lyrics as shown above. In contrast, Prosody and Explainable Prosody use prosody templates that explicitly relate lyrics to their rhythmic attributes (prosody). This strategy refines the original melody representation, guiding LMs to learn the rhythmic pattern of lyrics and effectively use the prosody information during inference.

Musical Prosody Faithfully Explains Lyrics Singability

To investigate the faithfulness of musical prosody to lyrics singability, we perform a correlation test between prosodyBLEU and human singability rating. The test results in a Pearson correlation coefficient r = 0.77 and P < .001, indicating a strong correlation. This suggests that musical prosody concurs with human perception of lyrical singability and constitutes a faithful singability explanation.

5.2 RQ2: Is It Useful To Communicate Musical Prosody as Singability Explanations to People? Singability Explanations Help Singers Quickly Align

Lyrics to Melodies

We analyse the average time (T) and average attempt number (A) that participants needed to successfully complete each singing trial. Figure 3 shows the aggregated T and A from all participants. Both T and A exhibit a decreasing trend from Vanilla to Prosody to Explainable Prosody, indicating that participants tend to take shorter/longer time and fewer/more trials to sing more/less prosodic lyrics.

For statistical evidence, a Friedman test was performed to compare T and A across three models. The results indicate a significant difference, with FrT = 808.13, FrA = 355.10, P < .001. Then, a Nemenyi post hoc test was conducted with holm correction applied to pair-wise comparisons between models. The results show significant differences between all model pairs as detailed in Table 3.

We also summarise some findings from participants' responses in the post-study briefing.

Participants Can Feel the Connection Between Prosody and Singability but Cannot Clearly Define It

Before being told the definition of prosody, all participants stated that they could feel something related to singability.

"I cannot tell exactly why, but I just feel that the arrangement of words in these sentences seems more consonant with melodies." - P2

"I think there are some rhythmic features that make the lyrics sound natural." - P1

"I find that capitalised words always appear at strong beats." - P8

However, when asked to summarise their descriptions into a clear definition, only 2 participants with lyrics writing experience precisely used the term prosody. The feedback shows the necessity of communicating prosody explanations to people, especially those without lyrics writing expertise.

Participants Confirm the Usefulness of Explanations

All participants agreed that the explanations were helpful, though they could use explanations for different purposes.

"These explanations did help me quickly detect the rhythmic rises and falls. I needed not spend more time studying them before singing." - P1, singer

"Besides singing, I love reading these explanations to quickly check whether the lyrics are rhythmically aligned to melodies." - P14, songwriter

P4 could also use the explanation to slightly adjust unsingable lyrics to be singable.

"If you asked me why I felt this was not singable, I could not explicitly tell the reason without these explanations. But with musical prosody, I know that a stressed syllable is missing here, and that I can insert one to make this sentence singable to me." - P4, singer

Moreover, all participants said that they dared try singing lyrics directly with explanations, instead of repeatedly reading and singing as they did without explanations. We analysed their attempt numbers (A) before being normalised by the sentence length |L| and found that 85.71% (396 out of 462) test sentences were completed in one trial with Explainable Prosody (viz., XAI-Lyricist), in contrast to 27.06% (125 out of 462) with Vanilla and 58.44% (270 out of 462) with Prosody. These findings all support the usefulness of communicating musical prosody as singability explanations.

Individual Preferences for Explanation Demonstrations Can Differ

We also found that the individual preferences for presenting explanations often differed. Generally, the preference depends on the background and usage scenarios of participants. For example, participants with singing expertise said they preferred music sheet-styled visualisations:

"I always read sheets when singing, so it'd be better if you wrote everything as sheets." - P1, singer

"Maybe you could give us more options, such as sheets, other than the piano roll interface, cuz different people have different habits. " - P5, singer

Participants with songwriting expertise said they felt comfortable with both sheet music and piano-roll visualisations as in Figure 1. However, they added that they wanted to know more musicological details (e.g. the definition of prosody).

"I felt it ok to see the piano roll, but knowing how these explanations are defined could further convince me." - P7, a songwriter with a degree in musicology

By contrast, participants with less expertise in songwriting mainly prefer the piano-roll interface:

"I don't read western sheet music, so I prefer piano roll because it intuitively presents the keyboard, pitches, durations and lyrics." - P6

These findings suggest the importance of considering users' preferences and backgrounds in practical applications.

6 Conclusion

In this paper, we present XAI-Lyricist, which uses musical prosody to guide language models in generating singable song lyrics and providing singability explanations. We used both computational metrics and a human-subjects study to prove the effectiveness of musical prosody in (1) improving the singability of LM-generated lyrics; (2) faithfully explaining lyrics' singability, demonstrating the usefulness of communicating prosody-based explanations. XAI-Lyricist also offers valuable insights into the singability explanations of LM-generated lyrics, potentially fostering a more efficient and effective collaboration between humans and AI in the domain of lyrics creation. In the future, we aim to explore other finer-grained domain knowledge to improve machine lyric generators, meanwhile exploring the pragmatic applications that cater to both experts and amateurs.

Ethical Statement

This work involved human subjects in its research. All ethical and experimental procedures have been approved by the Departmental Ethics Review Committee (DERC), National University of Singapore.

Acknowledgments

We thank all anonymous reviewers for their valuable input. This work was funded by the Ministry of Education, Singapore, under research grant MOE-T2EP20120-0012. Finale Doshi-Velez was supported by the National Science Foundation under Grant No. IIS-1750358. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.