LivePoem: Improving the Learning Experience of Classical Chinese Poetry with AI-Generated Musical Storyboards

1 Introduction

Textbook reading has long been the dominant method for classical poetry education in Chinese-speaking communities [Zhao and Li, 2024a]. It typically employs plain-language interpretations to help learners understand the archaic vocabulary and poetic techniques that are less common in modern Mandarin [Zhao and Li, 2024b]. While textbooks effectively convey the meaning of classical poetry, research suggests that extensive text reading can lead to boredom [KrukMariusz, 2021], disengagement [Guthrie and Davis, 2003], and negative learning outcomes over time [Feng et al., 2013].

Supplementary materials are available at: https://github.com/ lqhac/LivePoem

To improve the experience of poetry learning, researchers in the learning sciences have explored alternative or auxiliary multimodal methods to textbook reading, such as listening to music [Tilwani et al., 2022], viewing pictures and videos [Perez and Rodgers, 2019; Tahmina, 2023]. These methods highlight the role of audiovisual media in fostering various language skills [Khasawneh, 2023], including speech fluency [Hwang et al., 2024], vocabulary acquisition [Mu˜noz et al., 2023; Pratama and Hadi, 2023], and imagery comprehension [Pujadas and Mu˜noz, 2023]. Moreover, music has emerged as another promising modality for language learning, as it relates to the neural processing of tones and vowels in classical Chinese poetry [Zhang et al., 2023]. This underscores the positive impact of music on language education.

Despite these potential benefits, creating such audiovisual media can be costly, time-consuming, and require specialised expertise for human workers. To address this challenge, we develop LivePoem, a generative AI system that converts poetry into musical storyboards-storyboards with background music-as audiovisual learning materials. A musical storyboard includes a chain of images that visualise the poem's content, accompanied by the singing of poem lines (as in Figure 1). With AI-generated musical storyboards, we aim to enhance the learning experience of classical poetry by offering a more engaging form of educational media.

The system includes two phases: (A) storyboard generation, and (B) poem-to-melody generation. In (A), we employ a pre-trained language model (LM) and a latent diffusion model (LDM) [Huang et al., 2023] to generate storyboards for poetry. The LM expands a poem into a script that describes the poem's content in plain language, transforming the connotative and poetic language into more understandable descriptions. This script then prompts the LDM to generate visualisations of the poem's content. In (B), we train a prosody-based melody generator with a Transformer model [Lewis et al., 2020]. The poem is scanned and converted into a prosody template, conditioning the Transformer model to generate a melody that rhythmically aligns with the poem. This ensures that the output melody is singable, matching the poem's syllabic and rhythmic pattern [Liang et al., 2024]. Finally, the generated music is automatically aligned with storyboards by grouping musical phrases and images according to poem lines, enabling synchronised playback.

We validated the system using standard computational met-

Training-free, Based on an LM and an LDM (Huang et al., 2023)

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI-25)

Special Track on Human-Centred AI



Figure 1: Example of a musical storyboard for a Chinese poem Jing Ye Si by Li Bai. A musical storyboard consists of a sequence of visual frames with a background melody. The storyboard is interpolated into smoother animation, with the melody synthesised as singing voices.

Cutting-Edge Technical Underpinnings for Musical Storyboard Generation

Huang, H., Feng, Y., Shi, C., Xu, L., Yu, J., & Yang, S. (2024). Free-bloom: Zero-shot text-to-video generator with llm A musical storyboard includes a chain of images (video) and the singing of poem lines. We thus divide musical storyboard generation into two core technologies: text-to-video generation, and melody generation from poetry.

LM-Language Model; LDM-Latent Diffusion Model director and ldm animator. Advances in Neural Information Processing Systems, 36. rics from image and music generation studies, demonstrating its ability to produce high-quality content. However, recognising that automatic metrics alone cannot fully capture the human learning outcomes and experience, we further conducted a within-subjects study with 25 non-native Chinese language learners. This study (1) examined learners' preliminary learning outcomes, engagement, and satisfaction with musical storyboards versus textbooks, and (2) collected their opinions on both learning approaches. The results show that musical storyboards retained learners' test performance while making the learning experience substantially more engaging and pleasant. We also analysed learners' feedback that exposed the benefits and limitations of the traditional textbookbased and multimodal learning approaches.

In summary, our work contributes the following:

1. The LivePoem framework for musical storyboard generation to support classical Chinese poetry learning;
2. A two-part human-grounded study evaluating the effects of AI-generated musical storyboards in poetry learning.

2 RelatedWork

Textbook-based and Multimodal Learning

From lectures and standard tests to self-learning, textbooks have beeing playing a central role in various learning scenarios. Despite their ubiquity and effectiveness, recent research has identified some limitations of textbooks [Zhao and Li, 2024b], such as their overemphasis on vocabulary form and accuracy [Brown, 2011], and the lack of a meaningful context [King, 2002]. Furthermore, textbook-centric classes may lead to teacher over-involvement and student under-involvement [O'Neill, 1982], resulting in boredom [KrukMariusz, 2021], disengagement [Guthrie and Davis, 2003], and negative learning outcomes over time [Feng et al., 2013]. To address these issues, multimodal learning materials-especially audiovisual media [Tahmina, 2023]-have been shown to complement or enhance textbook-based learning. For example, Lee and R´ev´esz find that textually enhanced video captions can facilitate the acquisition of grammatical knowledge [Lee and R´ev´esz, 2018]. Zhang et al. observe the benefits of musical training for language learning, as music audio input facilitates tone and vowel processing abilities of language learners [Zhang et al., 2023]. Hnatyshyn et al. use melody alterations to symbolise cancerations in DNA, showing that knowledge musification improves the pleasantness of learning, compared to reading text materials [Hnatyshyn et al., 2024].

Text-based Video Generation. Text-based video generation creates image frames from texts that describe the expected content in the resulting video, which is often modelled as generating temporally coherent static images iteratively. In this field, diffusion-based models [Song et al., 2021; Kim et al., 2024] have become popular due to their highquality synthesis and semantic controllability. This success extends to video generation, where a promising direction is tuning pre-trained text-to-image generators in zero- or fewshot settings [Xu et al., 2023; Clark and Jaini, 2024]. One of the most recent works, Free-Bloom, uses pre-trained large language models to generate frame-level text descriptions and latent diffusion models for frame generation, achieving strong performance without additional training [Huang et al., 2023].

Generating Melodies for Poetry. Poetry and melody have long been closely related, both sharing a rhythmic and musical nature. Research has shown that early Chinese poems were composed for singing [Hu, 2023], and that singing in return contributed to the emergence of diverse poetic genres [Minli, 2024]. Generating melodies for poetry can be viewed as a subtask of lyrics-to-melody generation. Early deep learning methods mainly utilise end-to-end models (e.g., LSTMGAN [Yu et al., 2021]) on paired melody-lyrics datasets, but the size and quality of such datasets do not suffice to make large models converge well. Some researchers address this problem by pre-training transformer models on unpaired datasets [Sheng et al., 2021], which has spurred a wave of using attention-based methods for melody generation (e.g., [Liang and Wang, 2024]). Recently, research has explored the interpretability of AI melody generators, examining melodylyrics relationships in attention-based models [Duan et al., 2023]. Liang et al. highlight the importance of prosody in singability, which measures the rhythmic compatibility between melodies and lyrics during singing [Liang et al., 2024].

3 LivePoem SystemArchitecture

3.1 SystemOverview

LivePoem is a generative AI framework that automatically creates musical storyboards from classical Chinese poetry. A

musical storyboard for a poem contains (1) a sequence of images visually depicting the poem's content, paired with (2) the singing of the poem lines. As in Figure 2, this framework includes two phases: (A) the storyboard generation, and (B) the poem-to-melody generation.

3.2 Phase A: Storyboard Generation

In the storyboard generation phase (top panel of Figure 2), a language model first expands an input poem into a script. This expansion converts the connotative and metaphorical poetic language into more straightforward, plain-language descriptions, helping generative models capture the meaning of classical poetry. Next, a latent diffusion model [Huang et al., 2023] is prompted by this script to generate a sequence of temporally coherent images as the storyboard.

Frame-Level Script Generation

Let X = {x1, x2, . . . , xn} denote a classical Chinese poem with n lines. A language model L is employed to automatically generate a corresponding script Y = {y1, y2, . . . , yn}, where each yi is a plain-language description of the poetic scene depicted in xi, for all i ∈ {1, . . . , n}. To obtain Y , a system prompt X0 is provided to prime the input poetry X, specifying the generation task and guiding the model to focus on scene interpretation and descriptive fidelity.

where [X0,X] denotes the concatenation of X0 and X.

Script-to-Storyboard Generation

Prompted by the descriptive script Y = {y1, y2, . . . , yn}, a latent diffusion model D generates a sequential storyboard S = {s1, s2, . . . , sn}, where si denotes the set of frames depicting xi. To ensure semantic and temporal coherence across storyboard frames, we incorporate attention shift and sampling strategies proposed in [Huang et al., 2023]. Formally, this generation process can be expressed as:

3.3 Phase B: Poem-to-Melody Generation

To synthesise a singable background melody for the input poem, we train a poem-to-melody generator that produces a melody sequence M = {m1,m2, . . . ,mn} of n musical phrases, where mi corresponds to the i-th line of the poem. The generation of M is guided by the prosody of X, as the prosodic melody-poem alignment ensures that the resulting melody is both musically coherent and rhythmically compatible with the poetic structure [Liang et al., 2024]. Technically, this phase (bottom of Figure 2) includes: 1) a poetry scanner R, which scans 1 a poem X to extract its prosody structure and create a melody template; 2) a prosody-to-melody generator F, which employs an end-to-end Transformer model [Lewis et al., 2020] to generate a melody that matches the format and prosodic pattern in the template.

Poem Scansion

In classical Chinese poetry, the prosody of a poem is related to the poem's form, which stipulates the metrical and tonal format during composition. The prosody of poems is set by counting and grouping the syllables (characters) along with caesuras (in-line pauses) and a long pause at the end of the line [Birrell, 2022]. For instance, a five-character rhythmic poem consists of five characters (ch) per line. Each line is commonly segmented as either [2 ch|2 ch |1 ch] or [2 ch|1 ch |2 ch], where the first syllable of each segment is typically emphasised, and the final syllable tends to be lengthened [Kuo, 1971]. Based on this concept, we design a poem scanner that derives a melody template by analysing poetic structure, segmenting syllables, and annotating each syllable with strong/weak, short/long, and a binary LC marker for the last character of each line.

For example, in 静夜思 (J`ıng Y`e S¯ı), the second line is segmented as: 疑 是|地上|霜 (Y´ı sh`ı|d`ı sh`ang|shu¯ang), with the underlined characters stressed. Additionally, 霜 (shu¯ang) is also the last syllable of this line and can thus be represented as <strong,long,true>. When input to the model, the embeddings of all three symbols are concatenated and linearly projected to the embedding dimension of F.

1 Scansion, or scanning, is the analysis of the metrical patterns of a poem by organising its lines into feet of stressed and unstressed syllables and showing the major pauses, if any.

Template-to-Melody Generation

The melody generator F aims to generate a sequence of melody notes given a melody template. Each melody note n corresponds to a single time step in the sequence and is symbolically represented as a concatenation of five musical attributes (Figure 2, mid-bottom area): bar number Barb, position in bar Posx, pitch value Pitchp, note duration Durd, and a phrase boundary indicator Phraseh denoting whether n is the last note in a melody phrase. All subscripts (b, x, p, d, h) represent the actual value of each attribute. This representation minimises the length of melody sequences, reducing training costs while preserving melody structure. The model is trained, validated, and tested on LMD 2 and POP909 datasets [Wang* et al., 2020] (8:1:1 split), with the following objective function:

where L denotes the total loss; CEα denotes the cross entropy loss for an attribute α.

To ensure the alignment between melodies and poems, a sampling strategy is performed on bar and position symbols during inference: A penalty of r = -108 is added to the logits of the bar and position symbols that (1) have been previously sampled or (2) contradict the prosody pattern of poetry. This strategy ensures the monophonicity 3 of melody and the prosodic match between melody and poem.

With the generated storyboard S and the melody M, the storyboard frames and melody phrases that depict the same poem line are grouped together as a musical storyboard MS = {{s1,m1}, {s2,m2}, ..., {sn,mn}}. This synchronises melody and storyboard, enabling simultaneous playback. For better presentation, we follow [Huang et al., 2023] to interpolate the generated storyboard frames, thereby creating smoother animations, and use X Studio 4 to synthesise the generated melody into a singing voice.

3.4 SystemValidation

We validated the LivePoem system against state-of-the-art image and melody generators using a poetry corpus published by the Ministry of Education in China (with 137 poems) 5 . For storyboard generation, we used FIFO [Kim et al., 2024] as the baseline. The CLIP scores [Radford et al., 2021], which measure the similarity between an image and a piece of descriptive text, were 0.30 for LivePoem and 0.28 for FIFO. For poem-to-melody generation, we selected SongMASS [Sheng et al., 2021] and TeleMelody [Ju et al., 2022] as baselines. We first measured the singability of melodies using ProsodyBLEU (PB) [Liang et al., 2024]. Then, we evaluated the music quality of these models against a test dataset with 379

2 https://colinraffel.com/projects/lmd/

3 A melody is monophonic if it consists of a single, unaccompa-

nied melodic line, with no overlapping or simultaneous pitches.

4 https://xstudio.music.163.com

5 http://www.moe.gov.cn/srcsite/A26/s8001/202204/

W020220420582344386456.pdf; pages 58-63

songs randomly sampled from the LMD and Pop909 datasets. The metrics included Pitch Count (PC), Pitch Class Transition Matrix (PCTM), Pitch Range (PR), and Note Length Transition Matrix (NLTM) [Yang and Lerch, 2020]. The results (Table 1) demonstrate the competitive performance of LivePoem. On all metrics, a higher score indicates better performance.

4 Experiment

The experiment investigated two research questions (RQs):

RQ1: Can musical storyboards retain textbooks' effectiveness in improving learners' comprehension of classical Chinese poetry?

RQ2: Do musical storyboards provide a more engaging and pleasant learning experience compared to textbooks?

4.1 Study Overview

To address both RQs, we conducted a two-part user study: The first part used standardised reading comprehension tests to measure participants' poetry understanding using musical storyboards and textbooks (RQ1). The second used the selfassessment manikin (SAM) and free-response questions to explore participants' learning experiences with musical storyboards and textbooks (RQ2).

4.2 Participants

We recruited 25 Chinese language learners from our institution's mailing list, following these criteria: (1) non-native Chinese speakers; (2) aged over 18; (3) interested in classical Chinese poetry; (4) are not hard of hearing or experiencing limited vision. Participants were aged 18-35 (14 men, 10 women, 1 undisclosed). They self-reported their Chinese language proficiency using the Inter-agency Language Roundtable (ILR) scale 6 , a validated standard used by federal agencies in the U.S. for grading language proficiency: one at Level 0 (no proficiency), 14 at Level 1 (elementary), and 10 at Level 2 (limited working) (M = 1.4, SD = 0.6).

4.3 Part I: Evaluating the Effectiveness of Musical Storyboards in Learning Classical Poetry

The first part employed a within-subjects design and included the following two test conditions::

Textbook (TB): Participants read textbooks to answer reading comprehension questions. The textbooks contained the author's biography, annotations of important words, the translation and background of the poem, etc.

Storyboard (SB): Participants viewed musical storyboards to answer reading comprehension questions. The storyboards included visualisations and singing of the poems.

6 https://www.govtilr.org/Skills/ILRscale2.htm

To select poems for the reading comprehension tests, we referred to a corpus published by the Ministry of Education (MoE) of China 7 , which lists all compulsory poems in the national curriculum. The corpus was categorised into four difficulty levels: 1st-2nd, 3rd-4th, 5th-6th, and 7th-9th grades 8 . Regarding the number of questions, our pilot test showed that four poems, each with five multiple-choice questions, were optimal given time and cognitive load constraints. Each question included a correct answer, three incorrect answers, and an "I don't know" choice. This question set took 30-60 minutes to complete, aligning with the standard format of MoE poetry comprehension tests. To control order effects, both test conditions and poem samples were computationally randomised [Davies et al., 2014]. Specifically, we randomly selected four poems, one from each of the four difficulty levels. Two poems were randomly paired with musical storyboards, the other two with textbooks. The order of the four poems was also randomised when presented to participants. Participants were reimbursed at USD 7.59 per half hour.

The reading comprehension question sheet was created with Qualtrics 9 and followed these steps:

1. Participant Consent and Background: Participants read an introduction, signed a consent form approved by the ethics committee, and provided demographic details. They self-reported their Chinese proficiency using the Inter-agency Language Round-table (ILR) scale 10 .
2. Task Familiarisation: Participants completed a familiarisation session that simulated the formal study. They first read a poem (excluded from the formal study) and answered two test questions. They then read the same poem again with textbooks and storyboards and answered the same questions.
3. Pre-test: Participants read four poems sequentially. For each poem, they first read only the poem, without additional materials, and answered all questions sequentially. This step assessed their initial understanding and controlled prior knowledge differences.
4. Post-test: Participants reread each poem with either its musical storyboard or textbook and answered the same questions. Throughout the study, each test question was presented individually on a separate page. After submitting an answer, participants were not allowed to revisit previous questions.
5. Experience Rating: After completing the test, participants rated their experience with both materials on the Self-Assessment Manikin, a widely used measurement assessing engagement and pleasantness of users' experience [Hnatyshyn et al., 2024; Robinson and Clore, 2002].

Two metrics were computed to measure participants' performance: (1) Accuracy: the percentage of correct answers in the test; (2) Improvement: the difference in accuracy between pre-tests and post-tests.

7 http://www.moe.gov.cn/srcsite/A26/s8001/202204/ W020220420582344386456.pdf; pages 58-63

8 Grades here refer to school years in the Chinese education system, viz., 年级(ni´an j´ı).

9 https://www.qualtrics.com

10 https://www.govtilr.org/Skills/ILRscale2.htm

4.4 Part II: Investigating the Learning Experience of Musical Storyboards and Textbooks

The second part qualitatively investigates participants' experience with musical storyboards and textbooks. Apart from their SAM ratings, participants answered free-response questions about their preferences, perceived effectiveness, and additional insights or suggestions they wished to provide. A thematic analysis [Braun et al., 2019] was performed to summarise key insights from their responses.

5 Results

We first address RQ1 by quantitatively analysing participants' performance in the reading comprehension tests. Next, we investigate RQ2 by examining participants' ratings on SAM and analysing their free responses to the interview questions.

5.1 CanMusical Storyboards Retain Textbooks' Effectiveness in Improving Learners' Comprehension of Classical Chinese Poetry?

To measure the effects of textbooks and musical storyboards on poetry understanding, we compared participants' test accuracy and improvement under both conditions. Specifically, we fitted a linear mixed-effects model (LMM) [Meteyard and Davies, 2020] to predict participants' improvement in test performance (Table 2). The model included fixed effects of test condition (TB vs. SB), pre-test accuracy (Pre-Acc.), language proficiency (Proficiency), the difficulty and order of test poems, and an interaction between condition and pre-test accuracy. Participant ID was included as a random effect.

The model intercept revealed a significant overall improvement in test accuracy (β = 0.96, z = 4.87, p < .001) when SB was the reference level. This also held when TB was set to the reference level (β = 0.97, z = 5.59, p < .001), indicating that the reading comprehension performance was significantly improved by both TB (pre-test: N = 100, M = .60, SD = 0.25; post-test: N = 100, M = .80, SD = 0.23) and SB (pre-test: N = 100, M = .65, SD = 0.20; post-test: N = 100, M = .75, SD = 0.12). However, the Condition predictor (Table 2, row 2) showed that the difference in improvement between TB and SB was insignificant (β = 0.01, z = 0.12, p = .90). These results suggest that musical storyboards can facilitate the understanding of poems and retain the effectiveness of textbooks (Figure 3(A)(B)).

Pre-test accuracy showed a strong negative association with improvement (β = -0.78, z = -5.68, p < .001), indicating that participants who initially scored higher showed relatively less improvement. Poem difficulty (β = -0.02, z = -1.56, p = .12), language proficiency levels, and the order of test poems (β = -0.03, z = -1.24, p = .22) were not significantly related to improvement. Additionally, the interaction between test condition and pre-test accuracy was not significant (β = 0.13, z = 0.87, p = .38), suggesting a consistent relationship between prior knowledge and improvement across conditions. Overall, these findings indicate that learners' test performance was similarly improved by TB and SB, with limited influence of other factors such as language proficiency, poem difficulty, and test order (all p > .05).

The estimated variance for participants' random intercepts was 0.01 (SE = 0.04), indicating minimal individual differences in improvement after accounting for the fixed effects. The assumption diagnostics of the linear mixed-effects model showed no major violations of normality (Shapiro-Wilk test, W = .98, p = .10) or heteroscedasticity of the residuals.

5.2 DoMusical Storyboards Provide aMore Engaging and Pleasant Learning Experience Compared to Textbooks?

To assess participants' learning experiences with TB and SB, paired t-tests were performed on their SAM ratings. The results (Figure 3(C)) show that musical storyboards received significantly higher ratings on pleasure (SB: M = 6.68, SD = 0.93; TB: M = 5.72, SD = 1.64; t(24) = 2.87; p < .01) and arousal (SB: M = 5.56, SD = 1.20; TB: M = 4.76, SD = 1.63; t(24) = 2.38, p < .05). No significant difference was observed for dominance (SB: M = 6.04, SD = 1.89; TB: M = 5.84, SD = 2.07; t(24) = 0.53, p = .30). These findings suggest that participants, as language learners, perceive musical storyboards as more pleasant and engaging compared to traditional textbook-based learning.

We then performed an inductive thematic analysis [Braun et al., 2019] on participants' responses (numbered P1 to P25). Two raters from the research group independently coded the responses and concurred on four themes: poetry understanding through multimodality, guided and open interpretation, clarity of information conveyance, and enhanced learning experience and complementary methodologies. The Cohen's kappa yielded κ = 0.94 on all themes (M = .93, SD = 0.13, max = 1.00, min = 0.62), indicating an almost perfect inter-rater reliability [Landis and Koch, 1977]. The following sections present findings from the thematic analysis.

Poetry Understanding Through Multimodality. Almost all participants (23/25) mentioned that their general understanding of poems was improved by musical storyboards. Their responses highlighted the role of visualisations and music in their poetry comprehension. Specifically, 20 participants said that the visuals helped them better understand the poems. For example, "It directly illustrates the meaning of the difficult descriptions" (P9). Some also found the background singing helpful, as P5 commented, "I can't read the words but if narrated to me I probably can understand it.".

Guided and Open Interpretation. More than half of the participants (14/25) mentioned that their understanding was guided by textbooks or musical storyboards. For example, "it (traditional learning) is more direct in conveying the message" (P15). P9 felt guided by musical storyboards: "It provides storyboards to help people visualise descriptions that are hard to understand.". Moreover, some participants said musical storyboards inspired their own interpretation and imagination. "(storyboards) help to paint the image and clarify the context of what the poem is describing, while still leaving enough room for interpretation by the viewer themselves" (P13). These responses suggest that while both approaches can guide learners' understanding, they focus on different modalities: textbooks rely on verbal explanation, whereas storyboards offer visual context without constraining open interpretation.

Clarity of Information Conveyance. 16 out of 25 participants felt that musical storyboards effectively communicated the meaning of poems. P20 commented, "as a beginner it (storyboard) was a lot clearer and easier to grasp the essence of the poem". Nine participants supported the clarity of textbooks. For example, "... words translate better to me than pictures and music in the context of better understanding the poems" (P12). Three participants expressed that both approaches were effective in conveying the meanings of poems. For instance, "Storyboard-based learning can be as effective as the traditional learning for people who find it hard to grasp harder concepts with complex vocabulary." (P23). These insights suggest that musical storyboards can clearly convey poetic meaning, while participants' preferences for different modalities during learning reflect their individual learning styles. Beginners or visual learners lean towards visual elements that simplify textual descriptions, whereas others who enjoy reading more may prefer textual explanations.

Enhanced Learning Experience and Complementary Methodologies. Regarding learning experience, some participants noted that compared to musical storyboards, textbooks tended to be boring, confusing, and distracting. For instance, "I find it much more engaging with the visuals and the audio in storyboards. When I am faced with a wall of text, I often get lost and forgot what I have read earlier" (P17). Even some participants who preferred textbook-based learning acknowledged that "it (textbook) was boring" (P3). These comments reveal the limitations of traditional textbook-based learning and underscore the greater pleasure and engagement of storyboard-based poetry learning. Furthermore, some participants suggested that combining textbooks and musical storyboards could be ideal. For example, "Ideally, a blend of both would be good." (P19). These comments indicate that a complementary approach of both textbooks and musical storyboards can accommodate diverse learning preferences and further enhance the overall learning experience.

5.3 Discussion: The Role of Different Modalities

From the responses in Part II of our user study, we found that multimodal materials enhanced both engagement and pleasantness in effective poetry learning. To further investigate the roles of individual modalities, we conducted a survey with 11 native Chinese speakers (5 men, 6 women; aged 25-57, M = 27.7, SD = 9.4; labelled A1-A11), all of whom had studied classical Chinese poetry extensively through formal curricula. Participants reviewed learning materials for two poems presented in five single modalities: (1) textbooks, (2) static images, (3) animation, (4) singing (audio), and (5) reading aloud (audio). They were free to review them at their own pace. Participants then commented on what they liked, disliked, and felt could be improved in each modality, comparing their effectiveness in supporting poetry understanding. Key insights from their feedback are summarised below.

Textbook Reading Was Considered the Most Effective Modality. Most participants (8/11) identified textbook reading as the most effective and widely used method, highlighting its accessibility (A5), comprehensiveness (A1), information density (A4, A6, A7), and depth (A6). For example, "It provides the most well-rounded, comprehensive information" (A1). However, the experience was often considered "boring" and "daunting for beginners" (A5). As A3 noted, "If I weren't familiar with Chinese or interested in Chinese poetry, it would scare me away". These comments suggest that, while offering in-depth explanations, textbooks can be less engaging and deterring for beginners.

Visual Materials With Motion Enhance the Engagement of Poetry Understanding. Six participants expressed a preference for animations, describing them as "intuitive" (A9) and "engaging" (A3, A4), with A7 noting, "Even without reading the poem, I would know the vibe of the poem at first glance". Seven participants felt that animations were more dynamic than static images, enhancing their overall engagement. As A10 explained, "Animations tend to be more dynamic and vivid". This shows that motion in visual materials can significantly increase engagement with poetic content.

Singing Enhances Pleasantness, While Reading Aloud Preserves Tonal Accuracy. Several participants noted that singing made the learning experience more pleasant, with A6

stating, "I love music. It enhances the pleasantness of listening to poetry". Some mentioned that singing motivated their learning, with A5 adding, "It makes me want to know more about the poetry". However, A3 and A7 expressed concerns that singing could sacrifice some tonal accuracy, as it prioritised musicality over the accurate tonal information in Chinese. In contrast, audio recordings of poetry reading can preserve the correct tones, but were often considered "boring" (A9) and "plain" (A8). This suggests that while both speech and music convey the poem's content, a balance of both-such as offering both reading and singing-can maintain tonal accuracy while enhancing the pleasantness of the experience.

These findings reveal a trade-off between comprehensiveness and experience in multimodal learning. Learners' preferences are shaped by their prior knowledge, learning styles and objectives. While textbooks provide rich content, they may deter beginners due to their information density and cognitive load. In contrast, multimodal materials, such as music videos, can enhance the learning experience but may lack sufficient depth for advanced learners. Therefore, it is essential that educators with access to AI tools like LivePoem carefully select materials that align with learners' profiles and set personalised instructional goals.

6 Conclusion

This paper explores the possibility of applying AI-generated audiovisual media to Chinese language learning. We specifically focus on AI-generated musical storyboards and their effects on classical Chinese poetry education. To this end, we propose and implement a new generative AI system, LivePoem, which automates high-quality musical storyboard generation. Through a human-subjects study with Chinese language learners, we demonstrate that musical storyboards significantly improve the pleasure and engagement of classical Chinese poetry learning, while retaining the learning outcomes of textbooks. Based on these findings, we recommend integrating both traditional textbooks and multimedia materials more frequently in Chinese poetry teaching to enhance learner engagement and effectiveness.

Ethical Statement

This work involved human subjects in its research. All ethical and experimental procedures have been approved by the Departmental Ethics Review Committee (DERC), National University of Singapore.

Acknowledgments

We thank all reviewers for their input. This work is funded by a research grant MOE-MOESOL2021-0005 from the Ministry of Education of Singapore.