Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Framework

1 Introduction

Evidence-based interventions refer to practices and treatments grounded in systematic research and proven effective through controlled studies [Rutten et al., 2021][Melnyk and Fineout-Overholt, 2022]. It emphasizes the use of evidence from well-designed and well-conducted research as the foundation for healthcare decision-making [Sackett, 1997].

* Corresponding Author.

Intervention discovery from scientific literature enables researchers to keep abreast of the latest advancements and facilitate valuable insights that can significantly enhance the healthcare quality [Usai et al., 2018][Wang et al., 2023a]. However, due to the labor-intensive nature of human review, only a small fraction of intervention knowledge is systematically collected and curated. In healthcare, one of the biggest challenges for healthcare providers is the efficient identification of relevant intervention evidence from an overwhelming body of research, highlighting the urgent need for automated knowledge extraction tools to streamline the process and enhance the accessibility of this valuable information.

In recent years, large language models (LLMs) have been employed to categorize research papers, extract key findings, summarize complex studies, and create conversational assistants for question-answering and note generation, showing their impressive ability in understanding and extracting valuable insights from text [Achiam et al., 2023][Li et al., 2024a]. Many studies have utilized LLMs to streamline various subtasks involved in knowledge graph construction, such as named entity recognition (NER), relation extraction (RE), event extraction (EE), and entity linking (EL) [Wang et al., 2023b][Zhu et al., 2024]. Some research has also explored the collaboration between LLMs and human annotators to improve annotation quality [Kim et al., 2024a][Wang et al., 2024][Tang et al., 2024]. However, extracting intervention knowledge from long-range, domain-specific literature remains a significant challenge. On the one hand, developing human-annotated datasets for training deep learning models in NER and RE tasks requires specialized domain expertise to accurately interpret the literature. On the other hand, mining knowledge from long-range documents is a great challenge due to the vast volume of content, the inherent ambiguity of natural language, and the individual bias of human interpretation [Ye et al., 2022]. Particularly in healthcare contexts, these challenges are further compounded by the need for specialized therapeutic expertise, labor-intensive manual annotation, and difficulties in maintaining consistency and scalability [Zhao et al., 2021]. In this context, LLMs offer a promising alternative through in-context learning, enabling scalable information extraction without the need for extensive humanlabeled data. Despite these advancements, fully automated knowledge graph construction remains a challenge, particularly when dealing with long-range documents. Most current knowledge graph construction approaches focus on short texts, leaving significant potential for further development in handling more complex, lengthy content.

In this paper, we address the challenge of automating intervention discovery via LLMs by formulating it as a prompt design and annotation scheduling problem with a predefined intervention ontology graph structure and designing a framework leveraging two LLM agents to iteratively enhance the annotation quality. We introduce a progressive ontology prompting (POP) algorithm that employs an outdegreeprioritized breadth-first search (BFS) across the intervention ontology to create a series of prompt templates and action sequences to guide the annotation process conducted by LLMs. To enhance the annotation quality, we propose LLM-Duo, an interactive annotation framework by leveraging the power of LLMs while addressing the limitations of LLMs. Particularly, it integrates two LLM agents working both collaboratively and adversarially to refine annotation generation.

To showcase the practical impact of our approach, we apply our method in a case study of speech-language intervention discovery. We conduct experiments to compare our intervention discovery framework with several advanced baselines including long context LLM (i.e., GPT-4-Turbo with 128k context window length), and RAG-based annotation chatbot with advanced prompting techniques including Chain-of-Thought (CoT) [Wei et al., 2022] and Self-Refine [Madaan et al., 2024]. The experimental results demonstrate that our method not only delivers more accurate and comprehensive annotations over these strong baselines but also significantly accelerates the intervention discovery process. Furthermore, through our framework, we successfully curate a speech-language intervention knowledge base, providing a valuable resource for the speech-language pathology community. To our knowledge, this is the first intervention knowledge base in the speech-language pathology field.

Related Work. Traditional approaches to automated knowledge discovery typically rely on pipelines to handle various NLP tasks such as named entity recognition, relation extraction, coreference resolution, entity linking, and event detection [Luan et al., 2018][Martins et al., 2019][Wei et al., 2019][Zhong et al., 2023][Laurenzi et al., 2024]. Recent advancements leverage LLMs to generate relational triplets in zero/few-shot settings for knowledge graph construction, achieving promising results [Wei et al., 2023][Sun et al., 2024][He et al., 2024]. Some studies [Zhang and Soh, 2024][Carta et al., 2023][Vamsi et al., 2024][Zhu et al., 2024] have further streamlined knowledge graph construction by breaking it down into distinct phases, enabling LLMs to infer knowledge graph schemas without relying on predefined ontologies. However, these methods are often constrained to short texts or have only been validated on tasks like entity and relation extraction using human-annotated datasets, such as DuIE2.0 [Li et al., 2019] and DocRED [Yao et al., 2019], without being proven effective in real-world applications. Moreover, domain-specific knowledge often exhibits complex patterns that cannot be captured solely through sentence-level syntactic structures. As a result, most existing approaches [Du et al., 2020][Rossanez et al., 2020][Alam et al., 2023] are limited to handling abstracts and fail to extract and summarize knowledge across long-range contexts.

2 Preliminaries

A knowledge graph (KG) is a semantic network structured as an ontology, consisting of concepts and their relationships in a clear, interpretable format at scale [Peng et al., 2023]. For intervention knowledge discovery from literature, LLMs can enhance this process by leveraging their capabilities to understand long-range text. This allows for transforming unstructured data into structured formats, and populating the intervention ontology to create the intervention knowledge graph.

In our methodology, the intervention KG ontology is crafted by domain experts, which can be represented by a directed acyclic graph (DAG) G = (E↪R↪F). Here E, R, and F are sets of concepts, relationships, and semantic triples respectively. F is a collection of triples (h↪ r↪ t) with a head concept h 2 E, a tail concept t 2 E, and a relation r 2 R [Gruninger, 1995]. To effectively instruct LLMs to extract intervention knowledge anchored to G automatically, the design of annotation prompts and the query sequences plays a crucial role. We thereby frame the problem of automated intervention knowledge discovery via LLMs as one of prompt design and scheduling, described by the following equation:

where f is a function that translates intervention KG ontology into a set of annotation prompts and query sequences for the LLMs. A common case of f is directly prompting LLMs to generate triplets in a zero-shot/few-shot manner by including the whole KG schema within the prompt such as the annotation methods used in [Mihindukulasooriya et al., 2023][Kommineni et al., 2024]. However, those methods generate annotations in one shot and ignore the importance of contextual correlations between concepts within their surrounding neighborhood, resulting in incomplete annotations.

3 Methodology

In this section, we first introduce the POP algorithm that converts an intervention KG ontology into a set of annotation prompt templates and query orders, then propose an interactive annotation framework based on two LLM agents to enable more convincing and accurate annotation generations.

3.1 Progressive Ontology Prompting

We develop a progressive ontology prompting (POP) algorithm that employs a prioritized BFS on the intervention ontology graph G(E↪R↪F) to generate a set of annotation prompt templates and query sequences for LLMs. In our algorithm, the prompt formulation and scheduling follow a progressive manner. As illustrated in Figure 1, the annotation process begins at a source node (i.e., a concept node that only has outgoing edges) and continues by traversing its neighboring nodes in the order of a prioritized BFS. To allow for quick accessing a large portion of the graph, we enhance BFS by sorting neighboring nodes based on their out-to-in ratio R(v), which is defined by:

Our algorithm selects the neighboring node with the maximumR value to visit in the next step. For instance, in the example of Figure 1, visiting the 'Patient' node before the 'Disorder' node provides more context for the 'Disorder' concept annotation. For each concept node v, we use the visited nodes within its k hop neighborhood as its context. The Promptv for annotating concept v is crafted based on its context and the completed annotations within that context. The action order Orderv for Promptv is determined by the sequence in which node v is visited during the prioritized BFS traversal.

Our algorithm first follows prioritized BFS traversal to capture the local k hop context and visit order of concept node v, then composes the annotation prompts Promptv based on its ontology substructure Nk(v) and completed annotations within its context, which can be expressed as follows:

, where ⊕ is the concatenation. Promptv represents a set of annotation prompts for node v, generated by applying completed annotations to the prompt template Tv. As illustrated in Figure 1, the prompt template Tv consists of two parts: 1) Question, an annotation question derived from the relationship between node v and one of its neighboring nodes u; and 2) Prefix, a description based on the k - 1 hop path of neighbor node u. We leverage few-shot learning to task LLMs in generating prompt templates.

3.2 LLM-Duo Annotation Framework

To guarantee the integrity and reliability of LLM annotations, we propose LLM-Duo, a dual-agent annotation system. The central idea of a multi-agent system is to employ combinations of LLMs that can converse with each other to collaboratively accomplish tasks [Wu et al., 2023]. Drawing inspiration from the multi-agent debate idea in [Kim et al., 2024b], we design a framework where agents work both collaboratively and adversarially to enhance the quality of annotations.

The architecture of LLM-Duo is shown in Figure 2, featuring two LLM agents: the explorer and the evaluator. The explorer is a chatbot performing annotation tasks using zeroshot question answering (QA). To break the context window limit of LLMs and ensure the generated annotations are faithful to the provided literature content, RAG is employed in explorer to reference relevant sources, reducing LLM hallucinations. To improve the accuracy and reliability of the annotations, the evaluator LLM is set up to review and validate the explorer's responses, ensuring higher-quality results.

LLM-Duo will be tasked with annotation prompts following the sequential order generated by the POP algorithm. During each annotation cycle for a specific concept node, when focusing on concepts that emphasize rationality (e.g., disorder, intervention efficacy), the explorer provides an answer and an explanatory rationale to the evaluator. The evaluator then reviews the reasoning and offers feedback. Based

<Rationality>

Question1: Does this paper present any therapy/ intervention/treatment for individuals having speech or language deficits?

Answer1: Yes

Reason1: The paper discusses the tests of intonation-based speech treatment for minimally verbal children with autism.

Question2: Your answer might be problematic. {#Reason1}. Correct your answer and give strong evidences.

Answer2: Yes

Reason2: the paper discusses two interventions for minimally verbal children with Autism Spectrum Disorder (ASD): Auditory-Motor Mapping Training (AMMT) and Speech Repetition Therapy (SRT).

Question1: Analyze the following text:

'Answer':{#Answer1}; 'Reason': {#Reason1};

Do you agree with the answer based on the provided reason? Give your argument. Answer1: Disagree

Reason1: The 'Reason' does not confirm that any particular methods are proposed or detailed, only that their effects are being tested.

Question2: Analyze the following text:

'Answer':{#Answer2}; 'Reason': {#Reason2};

Do you agree with the answer based on the provided reason? Give your argument.

Answer2: Agree

Reason2: The 'Reason' clearly supports the 'Answer' as it states that the paper discusses therapies/interventions for minimally verbal children with Autism Spectrum Disorder.

<Completeness>

Question1: One intervention studied in this paper is Auditory-Motor Mapping Training (AMMT). Can you describe this intervention in detail?

Answer1: SRT is a control treatment designed to improve speech production accuracy in minimally verbal children with autism spectrum disorder...

Question2: Re-answer my question and cover more aspects except from: {#Aspects1}.

Answer2: SRT is a control treatment designed to improve speech production accuracy in minimally verbal children with autism spectrum disorder... 【SRT involves a series of structured steps to

facilitate speech repetition and accuracy. These steps may include...】

Figure 3: Annotation examples of speech-language intervention discovery using the LLM-Duo framework.

on this feedback, the explorer either refines its answer or, if in disagreement, presents stronger evidence to defend its original answer and challenge the evaluator's judgment. For concepts that emphasize completeness (e.g., intervention procedure, therapy activity), the evaluator extracts the aspects covered in each round of the explorer's answer, combines them with aspects from previous rounds, and prompts the explorer to expand further beyond the newly integrated aspect collection. This iterative process continues until the annotations reach a consistent and comprehensive state. As the example shown in Figure 3, by facilitating interactive loops between two LLM agents, LLM-Duo enables more accurate and complete annotations.

4 Experiments

4.1 Implementation

For LLM-Duo, the explorer is a chatbot built on LLM with RAG, implemented with Llamaindex 1 framework. We use OpenAI 'text-embedding-3-large' 2 as the embedding model and set the chunk size to 256 tokens with an overlapping size of 128. Particularly, we use 'FastCoref' [Otmazgin et al., 2022] to process text chunks for coreference resolution before text embedding. Additionally, we include the document ID as metadata for chunks and apply a metadata filter in the chat engine to ensure that the explorer only answers based on the specific document being annotated. We use Chroma 3 as the vector database. We set the retrieval to be on the top 8 text chunks based on similarity scores reranked with SentenceTransformerRerank 4 employing the 'cross-encoder/msmarco-MiniLM-L-2-v2' 5 model in Llamaindex. The evaluator is an external LLM who does not share any document context with the explorer.

1 https://www.llamaindex.ai

2 https://platform.openai.com/docs/guides/embeddings/embedding-models

3 https://github.com/chroma-core/chroma

4https://docs.llamaindex.ai/en/stable/examples/node postprocessor/

SentenceTransformerRerank

5 https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-2-v2

• Question1: What aspects this text is describing?{#Answer1}
• Aspects1: Objective, Target Population, Approach, Control Group, Outcome Measures, Results, Limitations.

Question2: What aspects this text is describing?{#Answer2}. Integrate your answer with {#Aspects1}.

• Aspects2: Objective, Target Population, Approach, Control Group, Outcome Measures, Results, Limitati-ons, 【 Therapeutic Steps, Therapeutic Techniques, Feedback Mechanisms, Duration and Frequency, Multisensory Approach, Individualized Treatment, Maintenance and Generalization, Collaboration with Caregivers.】

4.2 Case Study: Speech-language Intervention Discovery

Speech-language therapy provides interventions for individuals with speech-language deficits, enhancing their quality of life across various life stages. When choosing an intervention, evidence-based practice (EBP) is attractive as it integrates research evidence from literature into the decisionmaking process to ensure high-quality patient care [Law et al., 1996]. Intervention research, especially studies that offer clear intervention frameworks and comprehensive case studies, are valuable references to guide EBP designs. Intervention discovery aims to extensively gather speech-language interventions from the literature corpus as references to facilitate EBP design. It involves identifying relevant studies and extracting essential features of interventions including target disorder, procedure, efficacy, case study, therapy activity, etc., which is extremely labor-intensive for human reviewers, highlighting the efficiency of automating knowledge discovery based on LLMs.

To verify the effectiveness of our method in a realistic scenario, we employ our framework in a speech-language intervention discovery setting. The ontology is shown in Figure 4. To enable a large-scale discovery, we cultivate a literature base including 64,177 papers within the domain of speechlanguage therapy.

4.3 Annotation Baselines

The core idea of our automated intervention framework is to leverage the POP algorithm for guiding the annotation process while utilizing LLM-Duo to refine initial annotations by incorporating external feedback from another LLM. Instead of setting up another LLM for evaluation, recent studies demonstrate that LLMs can engage in self-correction to enhance their responses autonomously [Liu et al., 2024][Li et al., 2024b]. Notable examples of this include Chain-ofThought (CoT) [Wei et al., 2022] and Self-Refine [Madaan et al., 2024]. We separately equip the explorer chatbot based on RAG with these two prompting methods for annotation and denote them as CoT-RAG and Self-Refine-RAG. Additionally, in LLM-Duo, a potential substitution of explorer is long-context LLM, which is capable of processing entire document tokens instead of chunking and retrieval with RAG. We refer to the LLM-Duo system as LLM-Duo-RAG when using explorer built on RAG, and as LLM-Duo-LongContext when utilizing long-context LLMs. Besides, we also compare with methods that directly input paper text to LLMs for zero-shot QA annotation without the evaluation feedback loop, including ShortContext LLM, LongContext LLM, OpenAI Assistant, and RAG.

4.4 Evaluation

In the experiment of comparing LLM-Duo with annotation baselines, we report six types of metrics: 1) Consistency Rounds (CR): the number of refine loops the method makes before the annotation generation achieving consistency; 2) Verbosity Index (VI): the number of covered aspects per 1k tokens in the annotations, which is an important metric for emphasizing content completeness; 3) Enumeration Quantity

(EQ): the number of items listed in the annotations (i.e., therapy activities, therapy goals.); 4) Faithfulness (Faith): the extent of the annotation faithful to the provided literature literature, which is measured by FaithfulnessEvaluator 6 of Llamaindex. 5) Accuracy (ACC): the percentage of correct annotations in all LLM-generated annotations. 6) Cover: the percentage of correct LLM-generated annotations to the total mentioned concept entities in the provided literature.

5 Results

In this section, we first provide a detailed evaluation of our progressive ontology prompting algorithm and the LLMDuo annotation framework. Then, we showcase the results of speech-language intervention discoveries using our automated intervention discovery framework.

5.1 POP Algorithm Study

Context Size Analysis. In the POP algorithm, the context size k determines the diversity and volume of information included in the intervention annotation prompt. To assess the impact of context size on annotation quality, we conducted experiments using various k values to generate intervention annotation prompts for LLM-Duo-RAG. Specifically, we annotate the 'participant' concept for the experiment, which was based on a random selection of 8 speech-language pathology literature.

The annotation accuracy is shown in Figure 5a. The results indicate that as the context size k increases, annotation accuracy improves significantly, suggesting that a larger context provides more informative prompts, thereby enhancing annotation quality. Moreover, GPT-4-turbo consistently outperforms GPT-3.5-turbo across all k values, demonstrating that more advanced language models can further improve annotation accuracy. Besides, we inspect the text chunks retrieved back by different informative annotation queries based on various k values. We report the range of similarity scores and token count distribution of retrieved-back chunks in Figure 5b. The similarity score represents the semantically relevancy between retrieved texts to annotation queries. The results show that for k = 1, the retrieved text chunks generally have low similarity to the query, and the token count decreases as the similarity score increases, leading to lower annotation quality. In contrast, higher k values, especially k = 2, yield more relevant retrievals. For k = 2, the token count increases with higher similarity scores, indicating that richer and more relevant content is captured, resulting in improved annotation quality.

Prioritized BFS Analysis. In the POP algorithm, we employ the out-to-in ratio to prioritize neighboring nodes during BFS-based prompt creation and scheduling. This strategy ensures that nodes with more outgoing edges are visited first, allowing them to provide more context for annotating other nodes. For example, one annotation sequence following prioritized POP over the speech-language intervention ontology is 'TherapyActivity'!'TherapyGoal'!'Disorder'. In this section, we compare the 'Disorder' annotation results using the POP algorithm with and without prioritization, as

6 https://docs.llamaindex.ai/en/stable/examples/evaluation/faithfulness eval

• (a) 'participant' annotation accuracy using LLM-DuoRAG with GPTs at k=1,2,3.
• (b) Token count distribution of retrieved-back chunks across varying similarity scores using 'participant' annotation queries of k=1,2,3.

Table 1: Comparison of annotation results with baselines using different LLMs.

well as one-shot annotation without using POP, where the entire KG schema is included on a single annotation prompt to extract all triplets. The results are presented in Table 2. We can observe that applying both the POP and prioritized BFS notably enhances annotation performance.

5.2 LLM-Duo with Baselines

In this section, we compare LLM-Duo with several advanced baselines using annotations of 8 randomly selected papers from our speech-language literature corpus. The evaluation focuses on three key dimensions: 1) Intervention Recognition (IR), identifying intervention entities within the literature; 2) Intervention Aspect Summary (IAS), annotating the key aspects (e.g., procedure, therapy activity, therapy goals) of the intervention by capturing and summarizing relevant information from the paper; and 3) Intervention Knowledge Completion (IKC), linking interventions to theme classes (e.g., speech awareness, speech articulation, comprehension, foundation skills, etc.) and setting concept nodes (e.g., home, healthcare facilities, schools, teletherapy, etc.). We use human annotators for the IR and IKC tasks to generate goldstandard results for comparison. In the IAS task, we only ask human annotators to tag relevant text fragments related to specific intervention aspects due to potential individual bias in human interpretation.

The experimental results are reported in Table 1. It should be noted that we implemented 'ShortContext' using Llama3-instruct-70b (FP16) and Mistral-instruct-8x22b models (INT8). However, directly prompting these models with full paper text fails to produce annotations in a zero-shot QA setting. Their generations do not align with the annotation questions. The results in Table 1 show that LLM-Duo-RAG outperforms all baselines. Although GPT4-turbo has a 128k context window length and is capable of generating annotations, its annotation coverage remains inadequate. Integrating it with the LLM-Duo framework can significantly improve both the accuracy and the comprehensiveness of the intervention annotations. Additionally, compared with simple RAG, self-correct prompting methods such as CoT and Self-

• (b) Number of discovered interventions for top 20 disorders.
• (c) Number of discovered interventions for different age groups.

Table 3: Intervention-disorder examples in our discoveries.

Refine can significantly enhance intervention annotations, but their performance is still worse than LLM-Duo-RAG. Instead of utilizing costly GPT models, LLM-Duo-RAG, which employs open-source models including Llama3-instruct-70b and Mistral-instruct-8x22b, can achieve comparable annotation quality.

5.3 Ablation Study

In our implementation, the RAG technique serves as the backbone of explorer. We employ 'FastCoref' for coreference resolution and rerank retrieved chunks by similarity score using the 'cross-encoder/ms-marco-MiniLM-L-2-v2 model'. This section presents ablation studies for both components. We report the accuracy of intervention recognition in this section. As shown in Figure 6a, the results demonstrate that removing these components significantly decreases annotation accuracy, showing the necessity of each module.

5.4 Speech-Language Intervention Discovery

Through our automated intervention discovery framework, we identified 2,421 interventions supported by case studies from 64,177 literature in the speech-language pathology domain. The statistics of discovered interventions are presented in Figure 6b and Figure 6c. More intervention examples are provided in Table 3. 19 clinicians and students reviewed our annotations through online Google forms. We have constructed the first intervention knowledge graph in the speech-language pathology domain, which will be made publicly accessible upon acceptance. This knowledge graph is anticipated to be a valuable resource for domain experts, facilitating evidence-based clinical decision-making, questionanswering, and recommendation systems, ultimately contributing to improved healthcare outcomes.

6 Conclusion

In this paper, we developed a novel LLM-based framework for automatic intervention discovery from literature, featuring a progressive ontology prompting algorithm and a dual-agent system. The proposed method achieves superior performance compared with advanced baselines, enabling more accurate intervention discoveries. Our approach is adaptable to various intervention ontologies in healthcare and offers practical value to improve healthcare quality.

Acknowledgments

The research was supported, in part, by the National AI Institute for Exceptional Education (NSF Award #2229873), Center for Early Literacy and Responsible AI (IES Award #R305C240046), FuSe-TG (NSF Award #2235364) and SaTC (NSF Award #2329704). The opinions expressed are those of the authors and do not represent the views of any sponsors.