A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning

1 Introduction

Misdiagnosis remains a critical challenge in healthcare, leading to significant patient harm and increased healthcare costs [Newman-Toker et al., 2024; Dixit et al., 2023]. In clinical practice, decision-making relies on integrating diverse information sources, yet existing AI-assisted diagnostic systems struggle to effectively process and reason across multiple modalities [Lee et al., 2021; Microsoft, 2024; Rao et al., 2024]. To address this, we present MedRAG [Zhao et al., 2025], a smart multimodal healthcare copilot equipped with powerful large language model (LLM) reasoning, designed to enhance medical decision-making through multimodal integration and knowledge graph (KG)-elicited reasoning.

Through interviews with healthcare professionals, we identified key requirements for an effective AI-driven diagnostic assistant. Doctors emphasized that an ideal system should incorporate three primary input modalities to comprehensively support clinical workflows [OpenAI, 2023; Amballa, 2023; Zakka et al., 2024; Wei et al., 2018; Ren et al., 2024]:

∗Corresponding Authors

• Non-intrusive voice monitoring - Seamlessly captures real-time doctor-patient conversations during consultations without disruption. This enables instant follow-up questioning and context-aware diagnostic recommendations to support decision-making [Belle, 2023].
• General medical queries - Allows doctors to interactively refine differential diagnoses, seek clarifications, and receive personalized treatment suggestions in real time. This supports flexible information retrieval for both clinical and patient-facing decision support.
• Electronic Health Records (EHRs) - Analyzes similar cases to provide reasoning-enhanced diagnostics and personalized treatment recommendations, ensuring data-driven support for complex decisions.

While retrieval-augmented generation (RAG) has been proposed for medical AI applications, existing heuristicbased RAG models often fail to differentiate between diseases with similar manifestations [Wu et al., 2024b; Guu et al., 2020; Edge et al., 2024]. Doctors noted that these models tend to generate vague or incorrect recommendations, lacking structured reasoning capabilities [Zelin et al., 2024; Li et al., 2023; Wu et al., 2024a]. To overcome this limitation, we introduce KG-elicited reasoning, a key technology in MedRAG that enhances diagnostic accuracy by integrating structured medical knowledge with patient data.

MedRAG systematically constructs a hierarchical diagnostic KG, capturing subtle yet critical diagnostic differences. This KG is dynamically queried based on patient-specific manifestations and integrated with retrieved EHRs, allowing the system to reason through uncertainties and generate precise, context-aware diagnostic suggestions. Additionally, MedRAG proactively proposes follow-up questions to refine ambiguous cases, further supporting clinical workflows.

We evaluate MedRAG on both public (DDXPlus) and private (CPDD) datasets collected from Tan Tock Seng Hospital of Singapore. The results demonstrate its superiority over existing RAG approaches in diagnostic accuracy, specificity, and reasoning-based decision support. Our demo highlights how KG-elicited reasoning transforms MedRAG into a powerful, intelligent, and adaptable healthcare copilot, capable of assisting doctors across diverse clinical scenarios.

each disease are decomposed into unique features [Zhao et al., 2017; Li et al., 2018]. Features, diseases, subcategories, and categories are structured as nodes to form an undirected KG. Further, we apply GPT-4o to augment the differentiation of similar diseases by expanding more unique features of each disease within each subcategory. Given an undiagnosed patient's manifestations, MedRAG identifies the most relevant subcategory, and triplets ⟨disease, relation, feature⟩ associated with the identified subcategory are gathered as contextual information to elicit the reasoning capability of the backbone LLM.

Retrieval-Augmented Generation

To provide backbone LLM with case-specific information and mitigate hallucinations in generated outputs, we apply the RAG method, retrieving relevant documents before generation. In MedRAG, we use the EHR database as retrieval documents, as EHRs are systematically collected and structured within hospital databases. When patients' disease manifestations are fed into MedRAG, the systemmeasures the semantic similarity between input information and EHRs using cosine similarity. The top 3 relevant EHRs are then selected to provide the contextual input for backbone LLM. OpenAI's textembedding-3-large API is used as the text encoder to generate embeddings for both input information and EHRs.

Proactive Question Generation

When monitoring a medical consultation, MedRAG evaluates whether sufficient information is available for diagnostic reasoning by analyzing the semantic similarity of the input data and determining whether some EHRs meet a predefined threshold. If it is insufficient, MedRAG identifies the most critical unmentioned disease features in the diagnostic KG to differentiate between similar diseases and formulates followup questions. Otherwise, MedRAG proceeds to generate diagnostic recommendations.

2 SystemDesign

As shown in Figure 1, MedRAG incorporates three different modes of input, a KG-elicited reasoning RAG module and four outputs. MedRAG can seamlessly support various opensource and close-source LLMs, ensuring high adaptability and easy deployment in medical settings.

2.1 Multimodal Input

MedRAG provides three input modes to accommodate different clinical scenarios, as illustrated in Figure 1. MedRAG monitors doctor-patient conversations during consultations without interruption using Google's Speech-to-Text API in real time [Google Cloud, 2025]. With one-click activation at the beginning of the consultation, MedRAG automatically handles information collection, analysis, follow-up questions and diagnostic recommendations, reducing doctors' cognitive and operational workload while allowing them to focus on patient interaction. Doctors can also upload existing files like undiagnosed EHRs or simply type questions by keyboard. All collected information is processed by the KG-elicited Reasoning RAG for further diagnostic analysis.

2.2 Knowledge Graph-elicited Reasoning RAG

Knowledge graph-elicited reasoning RAG serves as the core analytical module of MedRAG, it constructs a diagnostic KG from the existing EHR database and identifies the most relevant subgraph based on the patient's manifestations [Yang et al., 2025; Lind et al., 1985]. It elicits the reasoning ability of the RAG model by extracting relevant triplets as context, which are then fed to the backbone LLM along with retrieved relevant documents, enabling more accurate and structured diagnostic insights.

Diagnostic Knowledge Graph

Given the EHR database, MedRAG constructs a four-tiered diagnostic KG by clustering diseases with similar manifestations into hierarchical categories while manifestations of

3 User Interface and Evaluation

We present the user interface (UI) and conduct comprehensive evaluations, including a case study and expert evaluation by doctors, to assess the system's performance.

3.1 UI of MedRAG

We provide a user-friendly interface built with Streamlit and CSS as shown in Figure 2. The left panel displays the chat history, allowing quick access to past consultations. The main panel at right presents three input modes: Speaking, Uploading Files, and Typewriting. The bottom section provides a text input field for direct user queries.

3.2 Case Study

In Table 1, we compare MedRAG with other LLMs including Llama3.1-8b, Mixtral-8x7b and Qwen2.5-72b, which often provide incorrect or ambiguous diagnoses, such as sciatica or radiculopathy, and struggle to distinguish similar conditions. In contrast, MedRAG accurately identifies lumbar canal stenosis and proactively generates follow-up questions to help doctors further refine the diagnosis.

Furthermore, a detailed demonstration scenario of a medical consultation with the corresponding suggested follow-up question, along with an end-to-end evaluation including voice modality, are provided in Appendix A-B 1 and Table 2.

3.3 Doctor Evaluation

To complement quantitative benchmarks with clinical insights, we incorporated a human evaluation involving four experienced doctors. These experts' feedback provides essential perspective on how MedRAG is perceived in clinical contexts, particularly in terms of trust and usability.

1 https://github.com/SNOWTEAM2023/MedRAG/tree/main/appendix

For the evaluation, doctors assessed three representative test cases with responses from both MedRAG and GPT-4o, focusing on functional design, user interface, EHR analysis, and medical consultation analysis. Our evaluation uses five Human Factors criteria (e.g., Clinical Relevance and Trust) widely used to assess AI-assisted systems [Choudhury and Shamszare, 2023; Choudhury, 2022; Zhao et al., 2021]. The details of the criteria definitions and specific questions are provided in Appendix C 1 .

The comparative results are presented in Figure 3. The results demonstrate that MedRAG outperforms GPT-4o across all criteria, with particularly outstanding performance in Adoption Intention. Some doctors emphasized that, since evidence-based practice is fundamental to medicine [Prasad and others, 2014], MedRAG stood out for its strong emphasis on evidence-based reasoning.

4 Conclusion

MedRAG is a smart multimodal healthcare copilot with powerful LLM reasoning, integrating multimodal inputs and KGelicited reasoning to enhance diagnostic accuracy and decision support. The results of case studies and doctor evaluation have consistently demonstrated the effectiveness and reliability of MedRAG in medical decision-making contexts.

Acknowledgments

This research is supported by the Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY) and the College of Computing and Data Science (CCDS) at NTU Singapore. It is also partially supported by the Singapore Ministry of Education Academic Research Fund Tier 1 (Grant No. 2017-T1-001-270). This research is also supported, in part, by the National Research Foundation, Prime Minister's Office, Singapore under its NRF Investigatorship Programme (NRFI Award No. NRF-NRFI05-20190002). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore. This research is supported, in part, by the Singapore Ministry of Health under its National Innovation Challenge on Active and Confident Ageing (NIC Project No. MOH/NIC/HAIG03/2017). This research is supported, in part, by the RIE2025 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) (Award I2301E0026), administered by A*STAR, as well as partially supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL). This work is partially supported by the Wallenberg Al, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Contribution Statement

Xuejiao Zhao and Siyan Liu made equal contributions.