Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds

1 Introduction

Clinical reasoning is a pivotal process where healthcare professionals incorporate clinical evidence and medical knowledge to assess, diagnose, and decide on treatment for patients [Montgomery Jr, 2018]. It entails a series of cognitive tasks, including gathering patient information, formulating and evaluating diagnostic hypotheses, and making treatment decisions [Young et al., 2018]. Consequently, clinical reasoning requires extensive medical knowledge and rich clinical experience, setting high expectations for physicians. However, in low- and middle-income countries (LMICs), medical resources are often scarce, making it difficult to access high-quality medical care. Despite accounting for 90% of the global burden of disease, LMICs only contribute to 12% of global health spending [Gottret and Schieber, 2006]. Furthermore, in 98 countries across Asia and Africa, the population of physicians does not meet the minimum threshold required for achieving 80% universal health coverage, underscoring a critical shortage of qualified clinical specialists [Haakenstad et al., 2022].

The shortage of healthcare resources urges the emergence of automated approaches with reliable clinical reasoning capabilities to support clinical decisions. Recently, Large Language Models (LLMs) have shown great potential in the medical domain [Lee et al., 2023], such as medical education [Lee, 2023], online consultation [Wu et al., 2024c], and clinical report summarization [Nayak et al., 2023]. Encouragingly, several advanced LLMs qualified for the medical licensing examinations at high scores, such as Med-PaLM [Singhal et al., 2023], ChatGPT [Wu et al., 2024b], and GPT4 [Nori et al., 2023], indicating remarkable proficiency in medical knowledge and clinical case analysis.

Despite the notable capabilities in comprehending humans' intentions and generating coherent responses [Zhao et al., 2023], directly applying LLMs to the medical field has also raised concerns over the generations of incorrect knowledge and hallucination during clinical reasoning [Bernstein et al., 2023; Liu et al., 2024]. This primarily stems from these advanced LLMs being predominantly trained on generaldomain data [Wu et al., 2024a]. Lacking extensive training in domain-specific text, they fail to encode sufficient medical expertise and comprehend medical texts laden with specialized concepts [Li´evin et al., 2022]. Absent a solid foundation of medical knowledge, LLMs struggle to grasp the intricate medical context and identify critical concerns behind it, making it difficult to generate comprehensive medical inferences.

To tackle these challenges, we propose a novel framework In-Context Padding (ICP) to enhance the inference capacity of LLM in the context of clinical reasoning. ICP consists of four major steps: 1) ICP firstly extracts medical entities from the clinical context and the reasoning objective; 2) In cooperation with the knowledge graph (KG), ICP then infer relevant medical entities (referred to as knowledge seeds) which could be helpful in clinical reasoning; 3) The acquired knowledge seeds are padded to the prompt and used to guide the inference process of LLMs; 4) finally, LLMs generate the clinical reasoning results as well the detailed explanation of how this reasoning is conducted. Extensive experiments and analyses on two datasets highlight a significant improvement in both the accuracy and interpretability of LLM. The ICP framework incorporates KG and in-context learning of LLM to efficiently bridge the knowledge gap in medical scenarios, ensuring its broad applicability in specialized domains. In addition to clinical reasoning results, the proposed ICP also provides an explanation of the reasoning process. Overall, the contributions are as follows:

• We propose In-Context Padding (ICP) which enhances LLMs to conduct clinical reasoning. This is especially beneficial for less developed countries where highquality medical care is hard to access.
• We infer the knowledge seeds from context information which are used as anchors for LLMs to conduct reasoning. This helps to align the LLM generation with the clinical reasoning process of physicians.
• Experimental results on two datasets validate the effectiveness of the proposed ICP. In addition to the final answer, ICP also provides a description of the reasoning process, making it more transparent and understandable.

2 RelatedWork

2.1 LLMs inMedicine

As several advanced LLMs have passed the medical licensing examination, there is a growing interest among researchers to explore the deployment of LLMs in clinical environments. Bernstein et al.(2023) and Ayers et al.(2023) compared the responses of ChatGPT with medical experts to healthcare questions posed by patients, which indicated the LLM could potentially offer helpful suggestions across various patient inquiries. Moreover, the LLM's responses were also rated significantly higher in terms of both quality and empathy. For multi-modality tasks in medicine, ChatCAD [Wang et al., 2023b] was developed to incorporate the LLMs for an interactive computer-aided diagnosis of medical images. Jeblick et al.(2022) and Lyu et al.(2023) assessed the capabilities of LLMs in translating radiology reports into plain language to be easier understood by patients.

However, the LLMmay provide false knowledge and hallucinations due to the misleading brought by the training corpus [Bernstein et al., 2023; Zhao et al., 2023]. This makes them challenging to deploy in a clinical setting, which necessitates rich medical knowledge and experience to conduct rigorous reasoning. While several studies attempted to integrate knowledge to enhance performance in medical tasks [Wu et al., 2024b; Gao et al., 2023], they typically required high-quality, large-scale, and structured medical knowledge for retrieval, which in turn limits their application. Furthermore, many of them focus on accuracy, overlooking the desire to improve medical inference during problem-solving, a crucial aspect of integrating LLMs into clinical practice.

2.2 Reasoning with LLMs

Owing to the exceptional ability in text generation and incontext learning, Wei et al. (2022) elicited the LLM to generate a detailed reasoning process before offering answers to questions under a few-shot setting, called Chain-of-Thought (CoT). It enhanced both accuracy and interpretability in various reasoning tasks. Then, Wang et al.(2022)explored the potential of sampling multiple reasoning paths, combining them using an ensemble voting technique to bolster performance. Meanwhile, Kojima et al.(2022) introduced a simple prompt, "Let's think step by step", to prompt LLMs to elucidate their analysis and then conclude with the answer, without any manually crafted examples.

Addressing the challenges of complex multi-step reasoning, numerous novel frameworks were developed to enhance the logical reasoning ability of the LLMs. Yao et al.(2022) decomposed the entirety of task-solving into reasoning and acting steps to progressively complete it. Taking a different approach, the Tree of Thought (ToT) [Yao et al., 2023] empowered LLMs to consider multiple different reasoning paths and self-evaluate choices to decide the next course of action, achieving notable performance improvements in tasks like the Game of 24 and word-based games. Furthermore, the Graph of Thought (GoT) [Besta et al., 2023] formulates the reasoning process using a graph to enhance the problem-modeling capability of CoT. Nonetheless, these studies mainly concentrated on mathematical reasoning and word games. There is limited exploration in the medical domain, where problemsolving also requires rigorous reasoning.

3 Methodology

3.1 ProblemFormulation

In this subsection, we formulate the clinical reasoning task. We denote an instance of clinical reasoning with four elements: {Q,O,R,A}, where Q refers to the context information and the reasoning objective, O = {o0, ..., os} refers to s candidate options, A refers to the correct answer which is one option in O, and R refers to the detailed analysis to conclude A. Given Q and options O, conducting clinical reasoning can be formulated as estimating the probability of generating reasoning steps and then determining the correct answer P(R, A|Q, O). As shown in Figure 1, Q refers to 1) the conditions of this patient; 2) the reasoning objective which is to infer the most appropriate treatment; O refers to five candidate drugs for treatments: Artane, Levodopa, Selegiline, Amantadine and Bromocriptine; A refers to the best choice Levodopa; R refers to the description of reasoning process.

3.2 Overall Workflow

Figure 1 illustrates the overall workflow of ICP. The fundamental principle of ICP is to identify the most relevant knowledge seeds and incorporate them into the in-context prompt as potential anchors. These anchors guide the construction of a multi-step inference path. The process comprises four main steps: 1) Identification of medical entities to understand the context information and the primary object for reasoning; 2) Mining of the potential knowledge seeds by incorporating identified medical entities and a medical knowledge graph learned from historical cases; 3) Composition of the whole prompt for LLM, including task instruction, context, options, and mined knowledge seeds; 4) Guiding clinical reasoning with LLM via knowledge seeds, which serve as anchors to conduct relevant inferences and conclude the final answer.

3.3 Identification of Medical Entities

For each instance, we concatenate Q and O and then extract the medical entities, including disease, symptoms, drugs, and any medical concepts, which briefly but precisely represent the medical context and the reasoning target. For the training instances, we also extract the medical entities discussed in its detailed analysis R.

To effectively extract the medical entities, we harness the exceptional in-context learning capabilities of LLMs and fewshot learning. Specifically, we provide the LLM with detailed instructions and representative examples to demonstrate the extraction process. This approach eliminates the time-consuming and labor-intensive need for manual annotation and specific model fine-tuning, making it easily adaptable to various clinical questions [Tarcar et al., 2019; Agrawal et al., 2022]. We select Baichuan2-7B-Chat [Yang et al., 2023] to identify medical entities, which is an advanced open-source LLM and demonstrated excellent performance across multiple Chinese tasks. More details can be found in the code repository 1 .

3.4 Construction of Medical Knowledge Graph

In this subsection, we construct a medical knowledge graph, G(E, V ), to encapsulate the relationship among various medical entities, where V and E denote the sets of m nodes {e1, ..., em} and edges vij of G, respectively. Each node in this graph represents a medical entity. The directed edge vij indicates the likelihood of discussing ej within an analysis given the presence of ei in the question and options. It is mathematically represented as:

1 https://github.com/Dragon-Wu/ICP-for-Clinical-Reasoning

An effective approach to compute the value of edge vij is counting the instances of ei in Q or O and ej in the corresponding analysis R. Without any training, this process can be directly calculated based on a training dataset encompassing n instances:

where {Qk,Ok,Rk,Ak} denotes a training sample k (refer to section 3.1). f is defined as:

Notably, due to the inherent directionality of the edges, vij and vji delineate the inverse path, leading to distinct values. If no relationship is established between node ei and node ej , the ˆ vij is designated a value of -1.

Some common entities have been frequently mentioned in many questions. To enhance the specificity between associated entities, we implement a weighting scheme. The resultant weighted edge value, vij , can be formulated by:

Here, cj denotes the frequency with which entity ej appears in the analysis of the training dataset:

To represent the entities and edges associated with ei, we introduce the Vei ∈ V and Eei ∈ E to denote all edges sourced from ei and their targeted nodes.

3.5 Mining Potential Knowledge Seeds

The essence of clinical context information can be approximately captured by the identified entities within Q and O. We extract the entities X from {Q,O}. For each extracted entity xi ∈ X, the edges started from it Vxi and the targeted nodes Exi are identified. Consequently, the aggregated sets of connected entities and edges are:

For each targeted entity e ∈ EX and its corresponding edge v ∈ VX, the correlation of e to an entity xi present in the question or options is computed. This relevance is determined by locating e in Ex and subsequently discerning the position of v in the sorted Vx:

Subsequently, we accumulate the ranks of e concerning all entities in X to estimate the overall correlation of e to all entities mentioned in Q and O:

Finally, we prioritize the entities e ∈ EX based on their value of q(e,X) and pinpoint the top 10 entities as the knowledge seeds, which are most likely to be discussed in inferences.

3.6 Guiding Clinical Reasoning via Knowledge Seeds

After detecting the knowledge seeds, we incorporate them into the prompt to steer the LLM toward producing a reasonable analysis. Specifically, we append these identified knowledge seeds following the questions and options. We then instruct the LLM with the detailed description, "Here is a clinical question, please refer to the knowledge seeds related to question-solving, and analyze this question step by step." This ensures that the LLM focuses and deliberates upon these entities, resulting in more concrete inference.

While we underscore the significance of these knowledge seeds, the LLMmight encounter challenges in understanding or effectively using them. Moreover, certain entities may be solely related to the entities within Q and O but do not facilitate its solution. To efficiently guide the LLM in harnessing these knowledge seeds, we adopt a few-shot learning strategy. Representative samples in the format of {question, option, knowledge seeds, analysis, and answer} are provided. Consequently, these exemplars guide the LLM to recognize helpful seeds and progressively factor them into the inferences, yielding a reasonable analysis and correct answer.

4 Experiments and Results

4.1 Dataset

The Chinese National Medical Licensing Examination (CNMLE) 2 serves as the official certification examination for medical practitioners in China, akin to the United States Medical Licensing Examination (USMLE). It is a prerequisite for the candidates to have undergone five years of medical education, in addition to at least one year of assessed clinical practice. The objective of the CNMLE is to evaluate the practical abilities in a real-world clinical setting. Therefore, we built two clinical reasoning data sets from CNMLE. The dataset statistic is shown in Table 1.

CNMLE-Clinical

There are different examinations for different clinical disciplines, the first dataset CNMLE-Clinical focuses on CNMLE for clinical medicine. The CNMLE-Clinical evaluates four parts of medicine: preventive medicine, preclinical medicine, clinical medicine, and medical humanities, which cover over twenty distinct medical subjects. We gathered questions from past examinations and various reference books, accumulating a total of 15,255 questions. Out of these, we randomly selected 600 questions, consistent with the number in the official examination, to serve as our testing set, while the remainder were used as the training set. Each instance in CNMLEClinical consists of a question, five candidate options, the correct answer, and a detailed explanation for the answer.

CMExam

The second dataset, CMExam [Liu et al., 2023], is a more comprehensive collection encompassing the six types of medical licensing examination: clinical medicine, traditional Chinese medicine (TCM), integrated TCM and Western medicine, dentistry, public health, pharmacy, and traditional Chinese pharmacy. The CMExam incorporates 68,119 medical questions sourced from past examinations and medical books. To ensure consistency with the genuine examination, we included 57,565 questions that have five options, a singular correct answer, and an analysis exceeding 30 words. Lastly, we randomly selected 600 questions to serve as the testing set, while the remaining questions in CMExam were designated as the training set. As shown in Table 1, the analyses of CMExam usually have more details than CNMLEClinical, which contains 25-29 entities in analyses, significantly higher than CNMLE-Clinical.

2 https://www1.nmec.org.cn/Pages/ArticleInfo-13-10706.html

4.2 Settings

The GPT 3.5-Turbo is selected as the primary model to evaluate, which includes 175B parameters and drives the online ChatGPT. All tests were conducted by calling OpenAI's official API. Unless specified, all experiments used the same parameters and were tested with the same version of the model (gpt-3.5-turbo-0613). To make the responses more deterministic and repeatable, we set the inference temperature to 0 to conduct greedy decoding, which always chooses the token with maximum probability instead of sampling. We keep the maximum context length (including prompt and response) of GPT 3.5-Turbo (4097 tokens) to avoid the potential performance penalty due to response length. The rest parameters are all set to default.

4.3 EvaluationMetrics

We evaluate model performance using both accuracy and NLG metrics. The accuracy generally represents the model's ability to conduct clinical reasoning. To assess the quality of the generated analysis, we employ BLUE [Papineni et al., 2002] and ROUGE [Lin, 2004], which are commonly used in medical inference [Liu et al., 2023; Tu et al., 2023] or medical report generation [Liu et al., 2021].

4.4 Baselines

To fully reveal the performance of LLMs with ICP, we evaluate several competitive baselines as follows.

• PLM-Finetuned: These baselines involve pre-trained language models (PLMs) extensively fine-tuned on the training set, including BERT [Vaswani et al., 2017], RoBERTa [Liu et al., 2019], Bart-base [Lewis et al., 2019], Bart-large, and PromptCLUE [Zhang and Xu, 2022] (based on T5 model [Raffel et al., 2020]). Encoder-only models like BERT and RoBERTa are designed to output results directly, whereas encoderdecoder models such as Bart and PromptCLUE can generate detailed explanations. Experimental setup and results referred to Liu et al.(2023).
• LLM-Finetuned: These baselines include LLMs fully fine-tuned on the training set, including both general LLMs (LLaMA [Touvron et al., 2023] and ChatGLM

[Du et al., 2021]) and medical LLMs (Huatuo [Wang et al., 2023a] and MedAlpaca [Han et al., 2023]). They support the generation of inferences and final answers. Moreover, the medical LLMs have also undergone additional training on medical corpora and medical question datasets [Wang et al., 2023a; Han et al., 2023].

• LLM-Zero-shot: The Standard QA employs the standard question-and-answer prompting that instructs the LLM (GPT3.5) to respond directly to the predicted answer without any explanation, as "Here is a multi-choice question about medical knowledge, please output the correct answer according to the question." The Chainof-Thought (CoT) promoting was introduced by [Wei et al., 2022] and [Kojima et al., 2022], which guides the LLM to solve questions step by step and prompts it to generate a detailed analysis, as "Here is a multi-choice question about medical knowledge, please analyze it in a step-by-step fashion and deduce the correct answer."
• LLM-Few-shot: We further enhance the reasoning capabilities of LLM through Few-shot approach, leveraging the in-context learning ability of LLMs. Given the context length constraints of LLMs, we randomly select six examples across a variety of clinical question types, including clinical case analysis, understanding of clinical knowledge, and medical computation tasks.

4.5 Result

As illustrated in Table 2 and Table 3, the proposed ICP outperforms baselines in both CMExam and CNMLE-Clinical, which significantly improves both the accuracy and quality of generated analysis. This verified the identified knowledge seeds encompass crucial information and thus efficiently guide the LLM to generate a convincing inference. Concurrently, we observe a strong coherence between accuracy and NLGmetrics, which suggests that an improvement in the reasoning process aids in addressing clinical questions.

Effect of Generating Analysis. Compared to the QAbased method, the CoT-based method, although producing detailed explanations, often compromises the final accuracy. This is different from the findings in other domains where generating a detailed reasoning process prior to outputting an answer can substantially improve model performance, as seen in mathematical reasoning [Wei et al., 2022; Kojima et al., 2022]. This divergence in the medical domain might arise because of its intrinsic demand for precise medical knowledge. Without the guidance of reliable medical knowledge, CoT might inadvertently generate false knowledge or hallucinations, consequently impairing performance. Similarly, the previous evaluation of LLMs in medicine [Wu et al., 2024b] also verified such a phenomenon.

Effect of Knowledge Seed. In contrast to CoT, our introduced ICP framework also improves the quality of generated explanations. Under the zero-shot setting, ICP achieves an increment of 3.32 in BLEU-4 and 1.43 in ROUGE-L for CMExam. Similarly, for CNMLE-Clinical, the uplift is 2.73 in BLEU-4 and 2.78 in ROUGE-L, resulting in accuracy improvements of 2.16 and 5.33, respectively. In CMExam with zero-shot, the accuracy of ICP is slightly lower than QA. This can be attributed to the intricacies of questions in CMExam, which involve more disciplines and usually contain more entities in longer analysis than CNMLE-Clinical (see Table 1). The incorporation of Few-shot strategy enables LLM to fully harness its in-context learning ability, which efficiently learns the patterns for problem-solving from examples, thus consistently improving model performance. The ICP with few-shot adeptly utilizes the predicted knowledge seeds for inference and achieves the best performance.

5 Analysis and Case Study

5.1 Impact of the LLMSize

To investigate the efficacy and generalizability of our framework, we conducted a comprehensive performance analysis on LLMs of varying sizes.

Baichuan2-chat (7B). The Baichuan2-chat-7B (BC2) was chosen to represent small LLMs, with its 7 billion parameters making it suitable for deployment on common PCs. It is an open-source bilingual LLM (Chinese and English) and has exhibited fine performance across various tasks. We assessed the reasoning abilities of BC2 under both zero-shot and fewshot scenarios for CoT and ICP using CMExam. Given BC2's extensive training on Chinese data, it achieved excellent performance on CMExam, only slightly behind GPT-3.5 (7B vs 175B). In a few-shot setting with ICP, its performance was nearly on par (50 vs 51.33), with a ROUGE-L score closely matching GPT-3.5 (26.50 vs 29.70). Such consistent and substantial improvements in a 7B model further validate the superiority of our framework, highlighting the potential of applying it to more resource-limited regions.

GPT-4 (∼1.7T). Subsequently, we examinedGPT-4, which is unofficially estimated to possess around 1.7 trillion parameters, making it the most powerful model available in terms of both size and performance. We accessed the GPT-4 (version: 0613) via its official API with the same parameters as GPT-3.5. Due to the API access rate limitations, we randomly selected 50 samples for testing. The results underscored GPT4's exceptional capabilities, as it achieved an accuracy of 72% even in zero-shot scenarios. Moreover, ICP consistently enhanced the model's accuracy and inference capabilities. This improvement also proves that our method can further enhance the model's ability for larger LLMs.

5.2 Performance on Different Medical Disciplines

To explore the usability of the LLMs integrated with ICP in medical scenarios, we further investigate its performance across different medical disciplines, areas of competency, and clinical departments. The subsequent analyses are based on the result of ICP under a few-shot setting with GPT3.5 in CMExam, as shown in Tables 5, 6, and 7.

As presented in Table 5, For clinical medicine, LLM with ICP achieved a high score of 60.77, qualifying for this licensing examination. This implies that its proficiency in medical knowledge is comparable to that of one certified physician. Furthermore, it achieved the highest ROUGE-L at 32.78 and BLEU-4 at 29.09. However, GPT-3.5 encounters challenges in public health and preventive medicine, yielding the lowest accuracy of 30.77%. In contrast, BC2 achieved an accuracy of 50%. Given that GPT-3.5 is predominantly trained on English corpus, this disparity is primarily attributed to that it involves many Chinese healthcare policies and regulations, such as "What is the stipulated standard for total coliform in China's drinking water sanitation guidelines?".

To holistically evaluate a physician's capabilities, the examination integrates a variety of clinical questions, including an assessment of medical knowledge, disease diagnosis, disease treatment, and preventive medicine. As illustrated in Table 6, the LLM achieves an accuracy exceeding 52% in three areas. The disease treatment necessitates considerations of the main symptoms, disease history, current condition, and other factors, making it a more complex clinical reasoning. Therefore, more adaptations and improvements are required to cope with these questions.

Diving deeper into various clinical departments based on Table 7, it reveals that the model performance varies considerably across diseases, ranging from a low of 38.89% in TCM to a high of 71.43% in the clinical laboratory. This variation can be attributed to the inherent complexity differences among diseases; for instance, questions about the laboratory department primarily focus on the interpretation of biomarkers, offering a more concentrated context.

5.3 Error Analysis

To further investigate the role of the ICP framework in the reasoning process, we conducted an error analysis based on the results of ICP with Few-shot (GPT-3.5) in CMExam, as recorded in Table 8. We analyzed the metrics associated with the predicted knowledge seeds. Each ground-truth explanation contains nearly ten medical entities. When compared with our predicted set of ten knowledge seeds, it became evident that questions with correct answers often had better knowledge seeds, as indicated by higher recall, precision, and F1 scores. Furthermore, regarding the length of responses and NLGmetrics, we observed that the correct response typically has higher ROUGE and BLEU scores despite having a shorter average length compared to the incorrect one. Overall, this suggests that our designed ICP framework and mined knowledge seeds can guide LLMs to conduct focused discussions on potentially critical aspects for question-solving, thereby effectively enhancing the quality of generated analyses and improving their reasoning ability in the medical domain.

After reviewing the reasoning process, unlike the standard QA that directly provides an answer option, several analyses of CoT and ICP fail to conclude with an exact answer option, as highlighted in Figure 2. It suggests that there might be multiple correct answers or no correct option. While there are insightful discussions in the reasoning process, and some approximately reached the answer, the final answer is still uncertain due to the deficiency of medical knowledge or incomplete reasoning. This underscores the rigor and complexity of clinical reasoning, necessitating further exploration into integrating more knowledge or enhancing reasoning abilities.

6 Conclusion

In this study, we proposed a simple yet effective In-Context Padding framework that 1) identifies potential knowledge seeds using the medical knowledge graph; and 2) enhances LLM on clinical reasoning. Experiments have shown that our framework can significantly enhance the inference capabilities of LLMs. During the LLM reasoning process, the mined knowledge seeds effectively bridge the knowledge gaps between LLMs and the medical domain. Extensive ablation studies and error analysis have proven the robustness and generalizability of our framework. Our efforts aim to ensure that LLMs are effective and equitable in specialized domains, especially in healthcare, thereby promoting global health equity through artificial intelligence solutions.

Ethical Statement

While the explored clinical questions involve specific analysis of clinical cases, all cases have been anonymized and deidentified to ensure that no personal information is disclosed The primary objective of this research is to investigate the reasoning abilities of LLMs in the clinical domain. To that end, we opted for a comprehensive evaluation using clinical questions. However, the results and discussions of this research are purely for academic research and analysis and do not serve as actual medical suggestions. Therefore, there is no negative impact on human health.

Contribution Statement

Jiageng Wu and Xian Wu contributed equally to this paper. Correspondence to Jie Yang.