CompLex: Music Theory Lexicon Constructed by Autonomous Agents for Automatic Music Generation

1 Introduction

Generative artificial intelligence in music has seen remarkable progress in both academia [Copet et al., 2024; Bhandari et al., 2025; Hu et al., 2025] and industry such as Suno [Suno, 2025] and Udio [Udio, 2025]. These advancements highlight the growing potential of artificial intelligence in the arts and creative industries [Wei et al., 2023; Karystinaios et al., 2023; Han et al., 2024], offering transformative capabilities for music creation, production, and consumption.

Despite significant progress, the development of generative artificial intelligence in music still falls short of the significant achievements seen in natural language processing (NLP) [Wei et al., 2022; Farquhar et al., 2024]. This gap is mainly due to the limited availability of music data, especially compared to the super large-scale datasets used to train large language models (LLMs) [Briot et al., 2017; Ji et al., 2023]. For example, while LLMs can ensure grammatical accuracy through data-driven pre-training and prompt tuning, large music models (LMMs) face challenges in adhering to music theory using data alone. This discrepancy stems from the vast difference in data availability between language and music models. Billions of new text data points are generated daily, whereas only tens of thousands of new pieces of music, both original and adapted, are created. The limited quantity of music data remains a challenge, as the total number of musical pieces from various periods and styles throughout history amounts to only millions 1 , starkly contrasting the trillion-level text corpora used to train LLMs [Achiam et al., 2023]. This limited quantity of music data remains a key bottleneck in the progress of generative music models.

∗Corresponding author.

Data augmentation has been proposed as a way to expand music datasets and has achieved significant performance improvements [Shorten and Khoshgoftaar, 2019]. However, we observe a challenging cycle: effective data augmentation relies on high-performing models, but improving these models depends on high-quality data, which is difficult to generate. Specifically, high-quality music data requires efficient generative models, and if these models perform poorly, the augmented data may introduce noise, undermining further training [Shorten and Khoshgoftaar, 2019]. Additionally, human auditory perception is highly sensitive to small changes in timing, pitch, and volume [McDermott and Oxenham, 2008], leading to low tolerance for suboptimal augmented samples. This cycle affects not only music but also many other domains, such as medical imaging [Shamshad et al., 2023] and finance [Cao, 2022], which also struggle with limited data availability.

In this context, knowledge-informed approaches are essential for enhancing the performance of music generation models. Music theory can significantly improve various aspects of music, including musicality [Akama, 2019], melody smoothness [Wu et al., 2020], and overall structure [Dai et al., 2020].

1 https://open.spotify.com/

Several components of music theory have already been incorporated into generative models, such as music themes [Shih et al., 2022], motif-level repetitions [Hu et al., 2023], and phrase-level call-and-response structures [Hu et al., 2024]. These works highlight that music theory can indeed boost music performance, even when only a few pieces of musical knowledge are integrated.

In this paper, we aim to leverage comprehensive music theory in AI-driven music generation tasks, such as algorithmic composition and style transfer, which have traditionally demanded considerable manual effort using existing methods. Specifically, we introduce a novel automatic approach to music lexicon construction that generates the music compositional lexicon, CompLex (Figure 1). CompLex is a comprehensive lexicon consisting of 9 categories, 90 properties, and 37,432 lexical items, all derived from just 9 manually input lexical category keywords and 5 prompt templates, significantly reducing reliance on human labor. To facilitate this process, we present LexConstructor, a new multi-agent algorithm designed to automatically detect and mitigate hallucinations during the generation of property-value pairs in the lexicon. CompLex shows impressive performance improvements across three state-of-the-art text-to-music generation models, covering both symbolic and audio-based methods. We also evaluate CompLex in terms of completeness, accuracy, nonredundancy, and executability, confirming that it satisfies the essential characteristics of an effective lexicon.

2 RelatedWork

2.1 Automatic Music Generation

Automatic music generation has seen significant advancements in recent years, particularly with the development of systems like Suno [Suno, 2025], which generates highquality audio-based music from user-provided text input. In academia, models such as MusicGen [Copet et al., 2024]

are capable of generating high-quality audio-based music, while Text2Midi [Bhandari et al., 2025] specializes in creating symbolic-based music using textual descriptions. Despite these advances, the challenge of limited data in the music domain remains a significant obstacle. To address this, knowledge-informed approaches have proven to be highly beneficial, as they integrate specific aspects of music theory to enhance generation performance, even with limited data samples. Notable works such as Transformer VAE [Jiang et al., 2020], Melons [Wu et al., 2020], MusicFrameworks [Dai et al., 2021], and MeloForm [Lu et al., 2022] have demonstrated the impact of incorporating music knowledge into generative models. In this paper, we aim to further advance AI-driven music generation by leveraging comprehensive music theory, specifically through the construction of a detailed music theory lexicon.

2.2 Autonomous Agents

Recently, autonomous agents have garnered significant interest, with notable success achieved by LLMs [Wang et al., 2024; Durante et al., 2024]. The applications of autonomous agents have quickly diversified into various tasks, such as software development [Qian et al., 2024], biomedical discoveries [Gao et al., 2024] and science debates [Cobbe et al., 2021], and fields such as music [Tatar and Pasquier, 2019], psychology [Kovaˇc et al., 2023], and finance [Mayo et al., 2024]. The strengths of the autonomous agent approach-specialization, and robust collaboration-make it particularly suitable for scenarios that require the integration of diverse expertise and coordination of multiple processes. To enable effective collaboration among agents, researchers have developed various communication strategies. For example, a role-reversal strategy is employed to mitigate hallucinations during direct agent communication [Qian et al., 2024]. To enhance performance in open-ended discussion scenarios, User request

1. Lexical Category Determination

Role: Category Architect

Task: Determine lexical categories and

preprocess the reference dataset.

Category

Note

Chord

…

Extract_lexical_

category(dataset)

Stage I: Lexicon Outline Creation

2. Lexical ItemOrganization

Role: Item Builder

Task: Manage and organize lexical

items from reference datasets based

on their respective categories.

Extract_lexical_item

(dataset,category)

Category Item

Note C#6

…

Chord F#m

…

…

Figure 2: Overview of the LexConstructor algorithm, which includes two stages: Lexicon Outline Creation and Lexicon Content Generation. Stage I focuses on determining the structure of the lexicon, while Stage II populates it by assigning property-value pairs to items in all categories. The user provides the input request, and then five types of autonomous agents collaborate to create the lexicon.

a debating strategy allows agents to brainstorm and refine ideas through structured debate [Chan et al., 2024]. The use of autonomous agents for automatically creating a comprehensive and compact lexicon remains underexplored. Traditional lexicon development relies on manual curation, which is time-consuming. To address this, we propose an automated music lexicon construction algorithm using multiple agents, reducing the need for costly human labor.

3 ProblemFormulation

We introduce a structured lexicon to organize the vast, unstructured body of music-theory knowledge based on four key components: Lexical Category, Lexical Item, Lexical Property, and Lexical Value.

Lexical Category. A lexical category is a high-level concept in music theory that groups related items. Let C denote the set of all categories in our framework. Formally, we define:

where each cj ∈ C represents a distinct music-theory concept. Example categories include "Chord", "Key", and "ChordProgression", among others.

Lexical Item. A lexical item is a specific instance of a category. For each category c ∈ C, let I(c) denote the set of items within that category. Formally, we define:

where each ij ∈ I(c) represents a unique item under the concept c. For instance, under the "Chord" category, we might have "CMajor", "AMinor", etc.

Lexical Property. A lexical property is a named attribute or feature that describes an item within a category. For each category c ∈ C, let P(c) denote the set of properties associated with items in that category. Formally, we have:

where each pj ∈ P(c) represents a distinct property under the concept c. For example, the "Chord" category might include properties such as "Name", "Type", and "Quality".

Property-Value Pair. Each lexical property pj of a lexical item can have a corresponding value, forming a propertyvalue pair. For an item i ∈ I(c) within category c, we define the set of property-value pairs as:

where vj is the value associated with the property pj. For example, for the item "CMajor" in the "Chord" category, we might have:

V(C Major, Chord) = { (Name, C Major), (Type, Triad), . . . }.

Objective. Our objective is to design a model that can automatically identify and extract a comprehensive set of categories C, the items of each category I(c), the properties of each category P(c), and generate the corresponding propertyvalue pairs V(i, c) correctly from unstructured music-theory knowledge, and finally construct CompLex D based on a user request u.

4 LexConstructor

We propose LexConstructor, a multi-agent algorithm composed of several specialized LLM-based agents. Specifically, LexConstructor includes five distinct agent roles, with

3. Lexical Property Creation

Role: Property Designer

Task: Define and create

lexical properties for each

category and its

corresponding items.

Category Property

Note Name

…

Chord Name

…

…

each agent assigned specific roles and tasks, based on one prompt template. LexConstructor operates in two stages, as shown in Figure 2: Lexicon Outline Creation and Lexicon Content Generation. In the Lexicon Outline Creation stage, three types of agents collaboratively develop the lexicon's structural framework by determining the necessary categories, items, and properties. In the Lexicon Content Generation stage, two types of agents populate the lexicon by assigning property-value pairs to each item within all categories.

4.1 Stage I: Lexicon Outline Creation

Inspired by how humans construct a lexicon from scratch, we begin by creating an outline of the lexicon in Stage I. In this stage, LexConstructor focuses on establishing the foundational structure of the lexicon by completing three key subtasks: Lexical Category Determination, Lexical Item Organization, and Lexical Property Creation.

Lexical Category Determination

We employ a specialized agent, the Category Architect (Aarc), to perform this subtask. The main responsibility of Aarc is to determine lexical categories based on user request u and to pre-process the reference MIDI dataset Dref into text-based structured information for future subtasks. To facilitate this process, Aarc utilizes a predefined function Extract lexical category(Dref ), which extracts lexical categories from the MIDI dataset based on the predefined categories. The agent analyzes the dataset, translating the MIDI information into a structured, text-based format, which will be used for the next subtask.

The output of Aarc is a set of categories C, formally defined as: C = Aarc(Dref , u), where u is the user request and includes 9 category key words in the work, which are mood, genre, key, instrument, tempo, time signature, chord, and note. In addition the reference dataset Dref is translated into structured, text-based information.

Lexical ItemOrganization

The Item Builder agent (Abuilder) is responsible for organizing lexical items within each category identified by Aarc. Due to the limitations of LLMs in generating large, unique item lists efficiently, Abuilder leverages the pre-defined function Extract lexical item(c, Dref ) to assist in item extraction. Given a reference dataset Dref and a target category c, this function helps the agent identify and extract all relevant items for that category. Formally, this step is represented as: I(c) = Abuild(c, Dref ).

Lexical Property Creation

The Property Designer agent (Ades) is tasked with defining and creating lexical properties for each category and its corresponding items. Taking the generated categories and items from earlier steps, Ades defines a set of properties P(c) for each category. The result is a comprehensive set of properties that describe each category's items, formally expressed as: P(c) = Ades(c, I(c)).

To ensure that the lexicon outline is both compact and comprehensive, the aforementioned subtasks are executed iteratively until all agents determine that no further refinement is needed. This iterative process allows the agents to refine their outputs based on the results from subsequent stages.

4.2 Stage II: Lexicon Content Generation

While the outline of the lexicon can be determined through tool usage and by utilizing information directly from the reference dataset, property-value pairs require the exploration of complex knowledge relations that can't be easily extracted from the dataset or hard-coded into the function. This makes value generation more susceptible to errors, such as inaccurate values assigned to properties. Therefore, we use the LLM's expert knowledge to generate these values during this stage. To further mitigate hallucinations and ensure accurate results, we design a Question-Answering (QA) communication strategy among agents.

Lexical Property Value Creation

We design two different types of agents: the Supervisor Agent (Asup), which oversees the property value generation process, ensures consistency, and manages quality control, and the Value Explorer Agents (Aexp), which provide detailed information and populate property values through collaborative exploration. To handle the large number of items to be populated into the lexicon, we define a function Process One Item(I) that allows the agent to process one item at a time, thus avoiding memory constraints of LLMs.

Formally, for each item i ∈ I(c), the value V(i, c) is generated as follows: V(i, c) = Asup (i, P(c)) , where Asup takes one item i at a time and generates the value corresponding to the property P(c).

The entire lexicon population process involves iterating through all categories and items, which is represented as:

Question Answering Communication

The Supervisor Agent formulates targeted questions, denoted as Q, and poses them to K Value Explorer Agents. These agents then collaborate, brainstorming and providing answers to the questions. The Supervisor Agent evaluates the responses and generates the final property-value pair V(i, c). The process can be formalized as:

where:

• Q = {q1, q2, . . .} is the set of diverse questions generated by the Supervisor Agent focus on item i.
• Qj ⊆ Q is the subset of questions assigned to Aexp,j .
• rj is the response from ATexp,j to its assigned questions at iteration T .

4.3 Text-to-Music Generation Application

The lexicon can be applied in text-to-music generation by refining the user prompt. Given a user prompt u, the system queries the lexicon D to retrieve related items I(cu), where cu is the category corresponding to u. For each item ij in I(cu), we extract its properties P(ij), which are then used to find additional related items I(ck) in other categories, expanding the context. The music is generated by using the related lexical items I(cu) ∪ I(ck). For example, if u = "happy mood", items from the "Mood" category can be quickly retrieved and then related information such as tempo (120 bpm), key (C major), and chord progression ("I-IV-VV"), among others, can be further retrieved.

5 Experiment

In this section, we provide the experimental setup used to evaluate the performance of CompLex on text-to-music generation tasks and the performance of LexConstructor method.

5.1 Implementation Details

Reference Dataset: We select the MidiCaps dataset [Melechovsky et al., 2024] as reference dataset, which is derived from the Lakh MIDI dataset [Raffel, 2016], one of the largest open-source collections of symbolic music data. MidiCaps contains 168,385 MIDI samples.

CompLex Statistics: The Genre category has 42 items and 13 properties; Mood has 48 items and 9 properties; Key has 24 items and 13 properties; Tempo has 13 items and 10 properties; Time Signature has 86 items and 12 properties; Instrument has 101 items and 8 properties; Chord has 193 items and 8 properties; Chord Progression has 36,797 items and 12 properties; and Note has 128 items and 5 properties.

Language Models: We test two versions of LLMs: GPT3.5-turbo-0125 and GPT-4o. For lexicon content generation, we utilize GPT-4o, selected for its superior efficiency and larger context window, supporting up to 16,384 tokens.

Model Configurations: All agents in the system are configured with a temperature setting of 0 [Chan et al., 2024]. The agents are assigned specific roles and tasks, with 5 prompt templates where each agent role corresponds to one prompt template, as shown in the Appendix. The user inputs 9 specific lexical categories in the experiment for comparison. The loop in T continues until the agents reach a consensus, at which point no further information is added. For the QA conversation strategy, we utilize K = 3 value explorers to facilitate brainstorming, enabling a thorough evaluation and refinement of the lexicon entries. For data analysis in the function Extract lexical category(Dref ), we follow implementations in [Melechovsky et al., 2024].

5.2 Baselines

For evaluating CompLex in text-to-music music generation tasks, we assessed its performance using three models:

• Text2MIDI [Bhandari et al., 2025], a SOTA open-source text-to-symbolic music generation model trained on the MidiCaps dataset;
• MusicGen [Copet et al., 2024], a SOTA open-source textto-audio music generation model in academia (we used the large version);
• Suno [Suno, 2025], a SOTA black-box text-to-audio music generation model in industry.

These baselines represent the most advanced models in both symbolic and audio music generation.

To validate LexConstructor, we compared it against several baselines and state-of-the-art (SOTA) approaches [Islam et al., 2024], including both single-agent and multi-agent models:

• Direct Prompting, where language models generate the music theory lexicon without explicit guidance.
• Chain of Thought Prompting (CoT) [Wei et al., 2022] breaks down tasks into step-by-step subtasks, facilitating the handling of more complex problems.
• Analogical Reasoning Prompting [Yasunaga et al., 2024] directs models to draw upon relevant past experiences to solve similar issues.
• Multi-Agent Role Reverse [Qian et al., 2024], where agents reverse roles to collaborate on problem-solving.
• Multi-Agent Debate [Chan et al., 2024], where agents engage in debate to find the optimal solution.

5.3 EvaluationMetrics

Objective Metrics

To assess the performance of CompLex on text-to-music generation tasks, we evaluate the following aspects: Compression Ratio (CR): Measures the structure of the music [Chuan and Herremans, 2018; Bhandari et al., 2025]. Contrastive Language-Audio Pre-training (CLAP): Measures how closely the music aligns with the corresponding text caption [Bhandari et al., 2025; Copet et al., 2024]. Mood Accuracy (MA): Measures how effectively the generated music reflects the mood indicated in the textual description [Melechovsky et al., 2024]. Genre Accuracy (GA): Measures how effectively the generated music reflects the genre in the textual description [Melechovsky et al., 2024].

To assess the performance of LexConstructor and other baselines, we evaluate the effectiveness of the generated lexicon based on the following aspects: Completeness: Measures how thoroughly the generated lexical items in each category align with the actual items in music theory. The completeness score is averaged across all categories, with higher scores indicating a greater coverage of the relevant items. Accuracy: Measures the alignment between the generated MIDI pitch values and frequency values relative to their corresponding note names. Higher scores reflect better accuracy and adherence to established music theory. NonRedundancy: Measures the level of non-duplication in the generated lexical items, calculated as the ratio of unique items to the total number of items. Higher scores indicate fewer redundant entries. Executability: Measures the lexicon's ability to be loaded error-free. Higher scores indicate greater reliability and functionality.

Subjective Metrics

Human evaluators assess the generated music from the music generation model based on Relevance to Text Input (REL) and Overall Quality (OVL) [Kreuk et al., 2022; Copet et al., 2024] from 1 to 100 where we emphasize musicality in OVL. An online survey was distributed via social media, receiving 24 valid responses after excluding invalid ones (e.g., participants selecting the same option for all questions or submitting incomplete responses). All participants were fully informed about the purpose of the study and consented to the use of their data for research purposes. Detailed information on the design of the subjective experiment is provided in the Appendix.

6 Results and Discussion

This section presents the results and discussion of our experiments. First, we demonstrate the impact of CompLex on text-to-music generation tasks by measuring the quality of the output music. Next, we demonstrate the effectiveness of LexConstructor by evaluating the quality of the generated composition lexicon in terms of completeness, accuracy, nonredundancy, and executability. Additionally, we conduct an ablation analysis to validate the effectiveness of the multiagent design to reduce redundancy and the QA communication to mitigate hallucinations.

6.1 Effectiveness of CompLex

We evaluate the performance of CompLex by refining the user text input on text-to-music generation models. We compare three different knowledge enhancement methods: W/o Enhancement, LLM-Enhanced, and CompLexEnhanced, which use raw input without any refinement, LLMs, and CompLex to refine the user input. In these methods, the user specifies mood or genre requests.

Table 1 summarizes the performance of the three baseline models across these methods. The metrics include Compression Ratio (measuring structural compactness), CLAP (measuring alignment between text and audio), Mood Accuracy, Genre Accuracy, Relevance to Input, and Overall Musicality. Across all baseline models, the CompLex-Enhanced method consistently achieves higher performance in all metrics, highlighting the benefits of integrating a music theory lexicon for generating high-quality music. These improvements are primarily driven by the lexicon's ability to provide specific and structured guidance during the generation process.

The LLM-Enhanced method generally underperforms compared to the CompLex-Enhanced method, primarily due to the lack of domain-specific music theory integration. Additionally, Suno's performance shows improvements when the lexicon is incorporated, underscoring the importance of leveraging structured music knowledge to achieve improvement in both objective and subjective metrics. These findings demonstrate the critical role of CompLex in advancing AI-driven music generation.

6.2 Effectiveness of LexConstructor

Table 2 compares the performance of LexConstructor with other baseline models across two different backbones. The evaluation metrics include completeness, accuracy, nonredundancy, and executability. LexConstructor outperforms all baseline models in every metric, highlighting its superior efficiency in generating a music theory lexicon. In addition, the improvements demonstrate LexConstructor's ability to mitigate hallucinations. These gains are largely due to the multi-agent role-playing system, where each agent specializes in specific tasks, and the Question-Answering (QA) communication strategy, which effectively mitigates hallucinations during value generation. The multi-agent approach consistently outperforms single-agent models, underscoring the effectiveness of using multiple agents for complex tasks. Overall, LexConstructor shows strong potential for automatically constructing high-quality lexicons, with applications extending to other domains in the future.

6.3 Ablation Analysis

AblationAnalysis ofMulti-Agent: Figure 3 illustrates our model's efficiency in terms of non-redundancy, showing its performance while generating 1000 lexical items with property values. Single-agent models begin to produce redundant outputs after generating about 50 unique items, with the issue being even more pronounced in the GPT-3.5-Turbo model. The superior performance of our approach can be attributed to its unique design, which incorporates specialized tool usage and task-specific subtasks. By focusing on generating one lexical item at a time, our method avoids referencing previously produced content. While a few duplicates still occur, this is due to the probabilistic nature of GPT models, which sometimes generate content similar to prior outputs.

Ablation Analysis of QA Communication Strategy: Ta-

ble 3 shows different communication strategies among agents. It indicates that the QA strategy outperforms other communication strategies, achieving the best performance in mitigating hallucinations that occur during property-value creation. This is because the diverse questions cover different aspects of the concept, and the value explorers collaboratively exchange and refine their answers until reaching a consensus, leading to more accurate and coherent results.

6.4 Case Study on HallucinationMitigation

Figure 4 presents a case study illustrating how LexConstructor effectively mitigates harmonic hallucinations. The note information in CompLex is correct and complete while generated from LLM-generated lexicon is incorrect and incomplete.

Figure 4: A case study demonstrating LexConstructor mitigate hallucinations.

7 Conclusion

In this work, we introduce a novel automatic music lexicon construction model that generates CompLex, a comprehensive lexicon containing 9 lexical categories, 90 lexical properties, and 37,432 lexical items, all derived from just 9 manually input category keywords and 5 prompt templates. We also propose a new multi-agent algorithm, LexConstructor, that generates the lexicon and automatically detects and mitigates hallucinations during the generation of propertyvalue pairs for each lexical item. CompLex demonstrates significant performance improvements across three state-ofthe-art text-to-music generation models, encompassing both symbolic and audio-based methods. Furthermore, we evaluate CompLex in terms of completeness, accuracy, nonredundancy, and executability, confirming that it meets the key characteristics of an effective lexicon. CompLex can be applied to a broader range of tasks within the music domain, including algorithmic composition, and style transfer. Currently, CompLex focuses on the structure of music composition, in the future, we plan to expand its scope to encompass expressive elements of music performance.

Acknowledgments

This work is supported by the project Towards Nextgeneration Artificial Auditory System with Brain-inspired Technologies (C5052-23GF).