PoemDirector: A Multi-Agent Context-Adaptive Instructional Mode Selection Framework for Chinese Classical Poetry Video Generation

1 Introduction

In China's K-12 curriculum, classical poetry is crucial to students' aesthetic education and cultural inheritance [Liu et al., 2023]. However, in many under-resourced schools, poetry is frequently taught by rote memorization due to teachers' insufficient training or time constraints, preventing students from building a genuine understanding [Zhou et al., 2025; Zheng et al., 2023; Xie and Deng, 2023; Liu et al., 2020]. Instructional videos on web-based platforms have become a primary learning resource and can help reduce the urban-rural quality gap [Yanghuang and Huang, 2024]. Yet producing high-quality videos demands significant effort [Brame, 2016; Manallack and Yuriev, 2016]. Even a 5-10 minute animated micro-lesson frequently involves dozens of hours of work [Guy and McNally, 2022; Xu et al., 2025]. As a result, the supply of quality poetry video resources falls well short of curriculum demand, and students in under-resourced regions remain underserved. Figure 1 shows an example of an instructional video for classical poetry.

∗Corresponding author.

Recent advances in generative AI have made it increasingly feasible to automatically create videos that teach classical poetry. Existing approaches fall into two categories. Semiautomated pipelines split the production process into multiple stages, each handled by AI tools, and then require manual assembly to produce the final video [Weerakoon et al., 2024]. In contrast, fully automated methods can generate complete videos directly from poem titles with minimal human intervention [Xu et al., 2025; Kasneci et al., 2023].

However, existing approaches face significant limitations. Semi-automated methods heavily rely on manual labor, leading to high production costs, while fully automated generation often lacks systematic instructional design, resulting in unclear logic and shallow explanations. Specifically, current end-to-end frameworks generally lack well-defined instructional objectives, presenting disconnected explanations rather than an organized understanding process. Furthermore, they lack teaching intelligence with contextual awareness, often employing a single narrative style that fails to engage students. Finally, hierarchical explanations aimed at deeper understanding are missing; most systems concentrate on superficial translations, preventing students from forming a genuine aesthetic appreciation of classical poetry.

To address these issues, we propose PoemDirector, a multi-agent framework that combines context-adaptive instructional mode selection, hierarchical explanation, and endto-end generation. Inspired by real-world creative teams, PoemDirector employs a Director Agent as the cognitive core that analyzes the poem and produces a creative blueprint centered on educational objectives. The blueprint directs three specialized agents: the Visual Designer generates visuals, the Narrator synthesizes multi-role speech, and the Video Editor integrates multimodal assets along the timeline.

This work originates from a collaboration with a national K-12 educational platform partner, which identified scalable poetry-video production as a practical need. Frontline K-12 language teachers contributed classroom interaction scenarios for different poem types and co-designed the instructional mode library used by the Director Agent. Literary experts iteratively refined the evaluation rubric, and education experts conducted annotation quality control throughout evaluation. By combining pedagogical expertise with automated generation, PoemDirector aims to provide low-cost, high-quality instructional resources for under-resourced educational settings. Our contributions are summarized as follows:

• We propose PoemDirector, a multi-agent framework that integrates context-adaptive instructional mode selection, hierarchical explanation, and end-to-end video generation through multi-role cooperation.
• We establish an expert-informed multidimensional evaluation framework that assesses both instructional quality and poetic presentation in classical poetry videos.
• We demonstrate through comparative studies that PoemDirector outperforms commercial baselines and approaches human-crafted video quality, and we report low-cost deployment on a real-world platform.

2 RelatedWork

To contextualize our approach, we review two lines of research most relevant to the automatic generation of instructional videos for classical poetry: AI-driven video generation and automated educational content creation.

2.1 AI-Driven Video Content Generation

Text-to-video generation has advanced rapidly, driven by diffusion models and flow-matching-based frameworks [Lipman et al., 2023; Ho et al., 2020]. Representative systems such as Lumiere and Make-A-Video demonstrate strong potential in generating short, high-fidelity video clips [Bar-Tal et al., 2024; Singer et al., 2022]. However, two major limitations persist for practical instructional use. First, video duration is severely constrained. State-of-the-art models typically produce clips of about 10 seconds, or roughly 240 frames, due to the computational demands of training [Xie et al., 2025]. Second, content controllability remains challenging. Video generation models are highly sensitive to prompt quality, and prompt optimization strategies are rarely evaluated for their effect on final output [Wang and Yang, 2024; Cheng et al., 2025]. These constraints make current models unsuitable for directly producing multi-minute instructional videos with coherent pedagogical structure.

2.2 Automated Educational Content Creation

Recent work has explored using generative AI to automate instructional material development and multimedia educational tools. Advances in multimodal generation, such as unified audio-text frameworks that jointly model speech and language, have expanded the technical foundation for building such systems [Liu et al., 2026; Li et al., 2021]. Chen and Wu propose an automated framework that converts Tang poetry into multimedia videos through text comprehension, image creation, and video synthesis [Chen and Wu, 2024]. Other studies show that generative AI can improve instructional design and the quality of micro-learning resources [Bolick and da Silva, 2024; Moore et al., 2025]. However, existing automated frameworks face two persistent limitations. First, interaction modes are largely passive. Most systems generate narrated slides or voiceovers with no dialogic elements. Research in learning sciences has shown that dialogic and narrative approaches can enhance student engagement and learning [Chi et al., 2018; Kuhn et al., 2014]. Second, content generation is rarely integrated with pedagogically sound multimodal planning [Peng et al., 2019; Paramythis and LoidlReisinger, 2004]. Large language model (LLM)-powered agentic systems offer a promising direction for coordinating multi-step generation pipelines through role-based cooperation [Hong et al., 2024; Qian et al., 2024]. However, no existing framework integrates pedagogically grounded instructional planning with end-to-end multimodal video generation for classical poetry education. This gap motivates the multiagent design adopted in this work.

3 Methodology

This section presents the framework of PoemDirector, as illustrated in Figure 2. The Director Agent converts a poem title into a unified creative blueprint through contextualized, hierarchical script design (Section 3.3). The Narrator synthesizes multi-role speech (Section 3.4), the Visual Designer generates visual assets (Section 3.5), and the Video Editor integrates multimodal outputs along the timeline (Section 3.6).

3.1 Problem Statement

The goal is to automatically generate an instructional video y ∈ Y for K-12 students based solely on a poem title t ∈ T . Unlike current approaches that prioritize visual aesthetics or rely on fixed narration templates, we explicitly model instructional objectives to guide explanation depth, narrative structure, and multimodal presentation. We organize this process as a collaborative endeavor among specialized agents to ensure instructional coherence and progressive explanation.

3.2 SystemOverview

PoemDirector decomposes this task into a set of specialized agents collaborating on a unified creative blueprint. We introduce the creative blueprint space B, the audio asset space A, and the visual asset space C. The four core agents are represented as parameterized mappings:

where θD denotes the parameters of the LLM used by the Director Agent. θN represents the speech foundation model parameters for the Narrator Agent. θV contains the combined parameters of the large visual model and the low-rank adaptation (LoRA) components used by the Visual Designer Agent. θVE denotes the program configurations in the Video Editor Agent for timeline construction and asset arrangement.

3.3 The Director Agent

The main challenge in generating instructional poetry videos is designing a systematic blueprint that achieves clear educational objectives. Most existing methods focus on visual aesthetics but lack a central module to coordinate poem understanding, instructional planning, and video presentation. PoemDirector addresses this challenge by assigning the Director Agent as the central coordinating hub.

Information Completion and Structured Analysis

The Director maps a user-provided poem title to a uniquely identified poem and its supporting materials. Let the system's curated poetry knowledge base be:

where tk is the poem title, pk is the full poem text, metak contains meta-information such as the author and dynasty, and rk denotes supporting references such as annotations, translations, and cultural allusions.

Candidate Retrieval with Abstention and User Confirmation. Given an input title string t (possibly partial), the system retrieves all title-matched candidates fromD:

where Match(·) implements title-based matching including partial-title matching after normalization. The retrieved candidates are presented to the user as a list for confirmation. If no candidate matches, the system abstains and no downstream generation is triggered:

where Select(·) denotes user selection among the candidate list. Once the user confirms a unique poem, the Director retrieves the complete poem package as grounded evidence:

Structured Analysis. The retrieved evidence (p,meta, r) is then organized into a structured representation. The Director employs the frozen LLM to produce a poem profile:

where H(·; θD) is implemented by the frozen Director LLM with a prompt template. Here, P(p,meta, r) is a structured poem profile covering themes, imagery, emotions, key cultural allusions, and pedagogical highlights.

Instructional Modes and Contextual Scripts

The Director selects a presentation form that best matches the current poem and its context from a finite library of instructional modes M = {m1,m2, . . . ,mK}. Each mode mj specifies (i) a context type such as parent-child dialogue, teacher-student instruction, or peer discussion, (ii) a role configuration and interaction style, and (iii) a script structure template together with mode-specific constraints. The mode library was co-designed with frontline K-12 language teachers, who contributed classroom interaction scenarios suitable for different poem types.

LLM-Based Mode Selection. The Director receives the structured poem profile P(p,meta, r), the mode library M, and a natural-language description of each mode template. It then selects a single mode m∗ ∈ M that best fits the current poem:

where Hselect(·) is implemented by the Director LLM with a prompt template to select exactly one mode.

Template Instantiation. After choosing m∗ , the Director instantiates its script structure template Tm∗ to build an explanation skeleton, represented as an ordered set of segments:

where τℓ is the segment position in the narrative, fℓ is the pedagogical function label, and ∆tℓ is the expected duration. This skeleton provides a structural interface for hierarchical content planning and time pacing.

Three-Level Instructional Interpretation

The Director then incorporates a three-level pedagogical interpretation structure to coordinate progressive learning. This prevents the explanation from degenerating into a shallow, linear narration. Each segment receives an interpretationlevel label hℓ ∈ H, where

The label hℓ is assigned by the Director LLM based on the segment function fℓ and the poem profile P(p,meta, r):

where Hassign(·) is implemented by the frozen Director LLM with a prompt template that outputs exactly one label fromH. The script skeleton is accordingly updated as:

In this structure, the literal layer targets word-level and phrase-level understanding. The imagery layer focuses on key poetic imagery that conveys emotion. The thematic layer addresses cultural connotations and deeper themes.

Creative Blueprint Generation

The Director expands the skeleton with the hierarchical labels to create a creative blueprint:

Here, S denotes the dialogue script, in which each line carries a role identifier, an interpretation-level label, and an expected duration. L defines the scene timeline that maps script segments to scene types and target durations. Q specifies image and video prompts for each scene, covering scene-related elements such as characters and locations as well as style constraints. Together, these three components ensure consistency in instructional logic, narrative structure, and visual style.

First, the Director uses the LLM to expand each skeleton segment into multiple dialogue lines:

where ui is the text of the i-th line, ri is the speaking role identifier, ci ∈ H is the pedagogical interpretation level, and ˆ τi is the expected duration.

Next, the Director aligns script segments with scene types to obtain the timeline:

where zj is the scene type, ℓj is the set of covered segment indices, and Tj is the target duration interval.

Finally, the Director generates a visual prompt per scene:

where each (qimg j , qvid j ) specifies the scene-related elements (e.g., characters, locations) and style constraints for image and video generation. This design ensures consistency in instructional logic, narrative structure, and visual style.

Several design choices collectively reduce the risk of factual errors in the generated content. The curated poetry knowledge base contains over 1.5 million records covering poem texts, metadata, annotations, translations, and cultural allusions. The Director retrieves grounded evidence from this knowledge base before generating any explanation, and the system explicitly abstains when no valid title match is found (Eq. 1). Structured mode templates and hierarchical interpretation labels further constrain the output space, limiting the LLM's generation to pedagogically scoped content. These safeguards are consistent with the strong human-rated knowledge accuracy reported in Section 4.4.

3.4 The Narrator Agent

The Narrator Agent generates multi-character voice tracks from the creative blueprint to replace monotonous singlespeaker narration. It assigns a vocal configuration v(ri) to each role ri (e.g., teacher, child, or elder) and synthesizes speech segments using a speech foundation model:

This auditory differentiation helps students identify speakers and creates a more immersive learning environment.

3.5 The Visual Designer Agent

The Visual Designer Agent generates visuals through two specialized modules guided by the creative blueprint: a LoRA-based module for stylistic atmosphere and a layoutdriven rendering module for textual readability. For each scene j, it uses the prompt pair (qimg j , qvid j ) from the blueprint to direct the static composition and dynamic motion.

LoRA-Based Classical Visuals

Generic large vision models often struggle with the specific aesthetic requirements of Chinese classical poetry. We apply standard LoRA adaptation to bias the diffusion backbone toward a consistent classical visual style, using over 100 curated images of characters and scenes in the Chinese classical style produced by a commercial text-to-image platform 1 and reviewed by human annotators. This lightweight adaptation restricts the visual output to a consistent historical style without the high cost of full-model retraining. During inference, the agent generates a static keyframe Ij, which is then expanded into a video clip clipj by an image-to-video model:

Poem Layout Rendering

To avoid character distortion and layout confusion in end-toend generation, the poem text is decomposed into characterlevel display units with pinyin annotations and rendered by an explicit layout engine. This engine maps each unit to spatial coordinates and style attributes and produces a text video track with strict time alignment to the audio narration.

3.6 The Video Editor Agent

The Video Editor Agent constructs a multi-track timeline based on the scene plan L. It aligns video segments with the synthesized speech duration and integrates all tracks with a unified resolution and frame rate to produce the final video:

4 Experiments

4.1 Dataset

For evaluation, we create a dataset of 100 poems spanning 7 dynasties, 46 poets, 6 genres, and 11 thematic categories such as pastoral landscapes, frontier warfare, friendship and farewell, and seasonal imagery. Approximately 95% of the poems come from prescribed K-12 curriculum texts, and the final selection was constrained by the availability of matched handcrafted animation videos (HAV) examples. Each test item provides only the poem title as input to all methods, and no external hints are supplied at inference time to ensure a fair comparison. The full dataset, automated evaluation code, and prompts are available in our public repository 2 .

1 https://jimeng.jianying.com

4.2 Baselines

We compare our method with four baselines for Chinese classical poetry video generation. From TikTok/Douyin 3 , we collect three types of videos: explanatory short videos (ESV), poetry recitation videos (PRV), and HAV. ESV and PRV were selected using objective popularity indicators such as view counts and comment volume. HAVs are meticulously crafted by professional human teams and serve as a high-quality human reference. We also include Mootion 4 , a general-purpose AI video production platform that can directly generate a poetry explanation video from a poem title alone. All reported results are averaged over the full 100-poem test set.

We exclude general video generation models such as Sora and Kling as baselines because they require manual prompt engineering, high-quality reference photos, and scene-level editing, and do not satisfy the title-only, end-to-end, multiminute instructional setting used here. The human-generated baselines (ESV, PRV, HAV) serve as static reference data without any human alteration during assessment.

4.3 Evaluation Procedures

Evaluation Metrics

We developed six evaluation metrics through collaborative expert design and iterative refinement. Three literary experts first defined the initial metrics, then scored a sample set of videos to identify scoring biases, which were addressed through iterative revisions until high consistency was achieved. The final suite comprises depth of explanation (DoE), knowledge accuracy (KA), instructional clarity (IC), aesthetic coherence (AC), poetic imagery restoration (PIR), and emotional resonance (ER). The scoring rubric for all six metrics is presented in Table 1.

Annotation

Human evaluation was conducted by three trained annotators with backgrounds in teacher education and expertise in Chinese poetry. Prior to formal annotation, a calibration session was conducted with iterative practice scoring on 10 samples until inter-annotator consistency (Cohen's kappa) exceeded 0.65. As shown in Table 2, the final agreement ranged from 0.69 to 1.00. Education experts randomly re-evaluated 10% of the annotations after each round for quality control.

4.4 Overall Performance

The overall performance is reported in Table 2. Commercial baselines show clear weaknesses: PRV scores below 1.02 in depth and clarity, while ESV and Mootion achieve only moderate performance. HAV, representing professional human-crafted videos, leads in most metrics. Our method closely approaches HAV in depth of explanation (4.07 vs

2 https://github.com/ai4ed/PoemDirector

3 https://www.tiktok.com/explore

4 https://www.mootion.com/

4.72), knowledge accuracy (4.10 vs 4.19), and instructional clarity (3.90 vs 3.92). Notably, PoemDirector surpasses HAV in poetic imagery restoration (3.97 vs 3.90) and emotional resonance (3.83 vs 3.74), which we attribute to the Director's structured content analysis and the Visual Designer's LoRA-based classical visuals. The largest remaining gap appears in aesthetic coherence (3.49 vs 3.91), reflecting the inherent limitations of current diffusion models in maintaining visual consistency across multi-minute video sequences. The gap in depth of explanation (4.07 vs 4.72) suggests that human experts still provide richer cultural context than the current retrieval-augmented pipeline. Despite these gaps, PoemDirector achieves this performance through fully automated generation at a marginal cost of $0.35-$0.45 per video, whereas HAV requires professional teams and substantially greater production effort.

4.5 Automated Evaluation with LLMs

To explore scalable evaluation, we develop an automated evaluator using multimodal LLMs. Video narrations are transcribed with Whisper large v3 and evaluated by Claude Sonnet 4.5 5 for DoE, KA, and IC [Radford et al., 2023]. Uniformly sampled keyframes are assessed by Claude Sonnet 4.5 for AC and PIR. Complete videos are analyzed by Gemini 2.5 Flash for ER [Team et al., 2024].

5 https://www.anthropic.com/news/claude-sonnet-4-5

As shown in Table 3, our method achieves competitive performance across most dimensions, particularly in KA (3.452), IC (3.935), and AC (3.935). The pipeline correctly identifies quality gaps such as PRV's deficiencies (scores of 1.000). However, LLMs exhibit compressed scoring ranges compared to human evaluators. For instance, ER scores fall between 4.161-5.000, whereas the human range is 1.94-3.83 in Table 2. These systematic differences motivate our multisource validation study (Section 4.6).

4.6 Multi-Source Validation Study

To address the limitations of single-modality evaluation, we conduct a multi-source validation study using VideoScore and cross-source consistency analysis [He et al., 2024]. As shown in Figure 3, our method achieves the highest dynamic degree score (3.41), attributed to the Visual Designer's imageto-video generation pipeline, while HAV and PRV rely on static imagery with minimal temporal variation. Figure 4

validates poetic imagery restoration across four independent sources: human expert annotations, LLM semantic extraction, CLIP-based vision-language alignment, and VideoScore text-video alignment. Our method ranks highest across all four sources, particularly in human annotations and CLIP scores, providing convergent evidence that the pipeline captures authentic poetic imagery.

4.7 Case Study

We conduct a qualitative case study on the Tang Dynasty poem "Farewell to Da Dong" by Shi Gao, with a visual comparison across three methods shown in Figure 5. Our method generates a video with three key strengths: (1) contextualized instruction through narrative storytelling enabled by the Director's contextual scripts, as shown at t=02s, (2) culturally authentic Tang Dynasty visual styling throughout the video, and (3) faithful poetic imagery restoration guided by the three-level instructional interpretation, including a flying goose in snowy mountains at t=85s and a traveler pointing toward distant horizons at sunset at t=140s. In contrast, HAV shows simplistic visual design with flat colors and cartoonish characters (t=275s), while Mootion relies on generic pastoral scenes (t=02s, t=45s) with text-heavy presentation.

4.8 Deployment and Collaboration

Stakeholder Collaboration. This project was initiated under the AI-for-Education program of a national K-12 educational platform partner, which identified scalable poetryvideo production as a practical need. Frontline K-12 language teachers co-designed the instructional mode library by contributing classroom interaction scenarios for different poem types. Literary experts iteratively refined the evaluation rubric through pilot scoring, and education experts conducted annotation quality control throughout evaluation. The platform partner also provided deployment infrastructure and integration requirements.

Deployment and Cost. We developed a web-based interface where users select a poem title and receive an automatically generated instructional video with no manual editing required. The Director Agent uses the GPT-4o API, while the Visual Designer employs locally deployed LoRA-finetuned FLUX and FramePack-1 models [Batifol et al., 2025; Zhang et al., 2025]. Only the Director incurs external API charges, and the marginal cost is approximately $0.35-$0.45 per video. The system was deployed in December 2025 on the partner platform, demonstrating practical feasibility as a low-cost educational content production tool.

Reproducibility. Representative demo videos, the dataset, evaluation scripts, the algorithm overview, and the detailed evaluation rubric are publicly available at https://github.com/ ai4ed/PoemDirector.

5 Conclusion

In this paper, we introduced PoemDirector, a multi-agent framework designed to automate the generation of instructional videos for Chinese classical poetry. Systematic experiments show that PoemDirector substantially outperforms commercial baselines while approaching humancrafted video quality. The low-cost deployment on a realworld K-12 educational platform demonstrates practical feasibility for alleviating the shortage of quality poetry teaching resources in under-resourced schools.

This work has several limitations. Our evaluation relies on expert assessments rather than direct student learning gains, the framework lacks personalized adaptation, and it has only been applied to classical poetry. In future work, we aim to address these through controlled classroom experiments, cognitive diagnostic techniques for individual adaptability, and extension to other literary genres.

Acknowledgements

This work was supported in part by NSFC under Grant No. 62477025; in part by Beijing Natural Science Foundation under Grant No. L252010; in part by Key Laboratory of Smart Education of Guangdong Higher Education Institutes, Jinan University (2022LSYS003) and in part by the Guangdong University Student Science and Technology Innovation Cultivation Special Fund under Grant No. pdjh2026ak027.