HealthLens: A Natural Language Querying System for Interactive Visualization of Electronic Health Records

1 Introduction

Electronic medical records (EMRs) are central to modern healthcare, integrating patient data such as diagnoses, treatments, lab results, and clinical notes [Williams and Boren, 2008; Fleming et al., 2024]. However, the coexistence of structured data (e.g., lab values) and unstructured narratives presents challenges in extracting actionable insights, tracking disease progression, and evaluating treatment outcomes.

al., 2017] or ECharts [Li et al., 2018]. This technical demand, coupled with the need for deep familiarity with medical data structures, limits the adoption of visualization tools in clinical settings. Existing solutions focus on niche applications like genomic data exploration [Pandey et al., 2023] or epidemic tracking [Crisan et al., 2022], while falling short in supporting multimodal EMR analysis. Recent work has aimed to bridge this gap, with methods such as weakly supervised pre-training for clinical text [Cai et al., 2023], but these solutions often require manual coding and are not optimized for interactive visualization. The advent of natural language processing technologies has reshaped the way users interact with data visualizations [Zhang et al., 2024]. For example, systems like FeVisQA [Song et al., 2024a] [Song et al., 2025] and approaches integrating dialogue systems with data visualization [Song et al., 2024b] show promise in enabling interactive, natural language-driven visualizations. Additionally, methods like Prompt4Vis [Li et al., 2025] highlight the importance of schema filtering and example mining, yet these strategies have not been fully adapted to the complexities of EMRs. To address these challenges, we introduce HealthLens, an intelligent visualization system tailored for EMR exploration. Our approach eliminates the need for expertise in declarative visualization languages through two key innovations: (1) MedCodeT5, a domain-specific language model fine-tuned on medical coding schemas, and (2) a bilevel optimization framework that integrates large language models (LLMs) with visualization grammars. By processing natural language queries like "Show how blood glucose levels relate to insulin doses over the past year," HealthLens identifies relevant patient cohorts, processes temporal data, and generates annotated line charts with statistical summaries-no manual coding required. This system significantly reduces the technical barrier to advanced EMR analysis, enabling healthcare professionals to focus on clinical reasoning rather than implementation.

Data visualization offers a promising solution by translating raw EMR data into interpretable charts and dashboards [Song et al., 2022]. However, this process remains complex, requiring professionals to extract relevant data, select appropriate visual encodings, and implement visualizations using declarative languages like Vega-Lite [Satyanarayan et Our contributions include: (1) the first system that leverages LLMs to convert natural language queries into EMR visualizations, (2) a method integrating MedCodeT5 with LLM-based visualization planning for enhanced accuracy, and (3) a user-friendly platform that requires no specialized technical knowledge, making it accessible to healthcare teams.

large-scale benchmark dataset tailored for medical text-tovisualization tasks developed by us, MedCodeT5 demonstrates exceptional performance in generating accurate and contextually appropriate visualizations from complex medical queries.

Capabilities and Strengths. MedCodeT5 has the following capabilities and strengths:

• High Accuracy in Text-to-Vis Tasks: The model achieves state-of-the-art performance in Text-to-Vis generation by using multi-task learning techniques. It surpasses existing baselines like Seq2Vis [Luo et al., 2021], ncNet [Luo et al., 2022], and CodeT5 [Wang et al., 2021] in terms of both visualization accuracy and overall robustness.
• Adaptation to Medical Contexts: MedCodeT5 is uniquely suited for handling the diverse and intricate data types found in medical records. This domainspecific training allows the model to account for medical terminologies, abbreviations, and context-sensitive data relationships, which are often critical in healthcare scenarios.
• Robust Performance Across Difficulty Levels: MedCodeT5 demonstrates superior performance across varying levels of query complexity, from straightforward requests to highly intricate queries involving multiple clauses, nested conditions, and advanced filtering.

2.3 Auto Refinement and Correction of DVL

HealthLens features a robust error-handling mechanism powered by advanced large language models (e.g., GPT4), ensuring accurate visualizations even when user queries or generated specifications contain errors. When the MedCodeT5 model translates a query into DVL, it automatically detects errors during execution (such as invalid field references or unsupported chart types) and logs them. These error details are then sent to the LLM, which analyzes and refines

2 SystemDescription

To enable users to effortlessly create visualizations of EMR data using natural language instead of DVLs, HealthLens leverages MedCodeT5 as its foundational model. This model translates natural language inputs into corresponding DVL specifications, simplifying the visualization process for nontechnical users. Once the DVL specifications are generated, HealthLens executes them to produce the final visualization using the Vega-Lite framework, delivering high-quality and interactive graphical representations of the data. In cases where the execution fails, HealthLens captures the error information and utilizes a large language model, such as GPT4o [OpenAI, 2024], to analyze the error, correct the DVL based on the feedback, and refine the previous specifications. This iterative process ensures robust and accurate visualizations, even in the presence of initial errors. This process is shown in Figure 1.

2.1 MedicalVis Dataset

The MedicalVis dataset forms the backbone of MedCodeT5's capabilities. Comprising 35,374 instances and covering a diverse range of visualization types-including bar charts, pie charts, line charts, and scatter plots, MedicalVis provides a comprehensive benchmark for medical text-to-visualization tasks. The dataset was constructed using a novel pipeline that leverages large language models to generate high-quality DVLs from structured SQL queries. By training on this dataset, MedCodeT5 not only learns to generate accurate visualizations but also gains a deep understanding of medical data semantics, query structures, and visualization aesthetics.

2.2 Text-to-Vis UsingMedCodeT5

MedCodeT5 is specifically optimized for generating DVLs in the medical domain, enabling healthcare professionals and researchers to intuitively interact with EMRs using natural language queries. By leveraging the MedicalVis dataset, a the DVL. This process iteratively suggests corrections until a visually effective and accurate visualization is produced.

The mechanism is divided into two levels: the low-level and the high-level. At the low level, the LLM uses multiturn interactions to correct errors in the query, thus improving model accuracy. At the high level, the LLM is used for chart optimization, such as applying raster graphics, adding color, or inserting textual explanations. The first stage is currently implemented using GPT-4 and prompt engineering; when errors occur, GPT-4 is invoked after execution to correct them based on the error information. For chart optimization, GPT-4 generates corresponding chart descriptions based on the metrics and can enhance the chart's appearance, including adding colors or tooltips on the chart.

3 Demonstration HealthLens

To demonstrate the capabilities of HealthLens, we provide an example application scenario where a healthcare professional utilizes the tool to analyze patient data from EMRs. This scenario highlights the seamless interaction between the user and HealthLens, showcasing its ability to generate visualizations from natural language queries.

3.1 Application Scenario

A hospital administrator, Dr. Smith, wants to analyze the distribution of patient lengths of stay in the hospital to identify patterns and optimize resource allocation. Dr. Smith is not familiar with DVLs like Vega-Lite but is comfortable describing their needs in plain language.

Step 1: Input Natural Language Query

Dr. Smith opens HealthLens and sees a simple, user-friendly interface (shown in Figure 2) with a text input box. Dr. Smith types the following query:

"Create a bar chart visualizing the number of patients for each hospital stay duration."

Step 2: Query Processing

Upon submitting the query, HealthLens processes the input using the MedCodeT5 model, which translates the natural language query into a DVL specification. This specification is then executed using the Vega-Lite framework to generate the desired visualization.

Step 3: Visualization Output

Within seconds, a bar chart appears on the screen. The chart displays the distribution of hospital stay durations, with the x-axis representing the length of stay (in days) and the y-axis indicating the number of patients. The chart is interactive, allowing Dr. Smith to hover over each bar to see specific counts.

Step 4: Refinement and Customization

Dr. Smith wants to refine the chart further to focus on stays longer than 10 days. They type a follow-up query:

"Filter the chart to show hospital stays longer than 10 days."

If the chart is generated incorrectly, HealthLens will update the visualization with LLM to output the desired result and apply the specified optimization scheme to produce the beautified chart along with an explanation. Dr. Smith can now better analyze this subset of data to identify potential bottlenecks in the patient discharge process.

3.2 Key Features Highlighted in the Scenario

• Natural Language Querying: The ability to translate plain language into sophisticated visualizations eliminates the need for technical expertise.
• Interactivity: Users can interact with and refine visualizations dynamically, enhancing data exploration.
• ErrorHandling andCorrection: In case of insufficient or ambiguous input, HealthLens offers intelligent suggestions to guide the user.
• Accessibility: The tool's intuitive interface ensures that both technical and non-technical users can benefit from its advanced visualization capabilities.

This demonstration underscores the transformative potential of HealthLens in simplifying EMR data visualization. By lowering technical barriers and enabling intuitive interaction, the tool empowers healthcare professionals to extract meaningful insights from complex data, ultimately contributing to better decision-making and patient care.

4 Conclusion

In this paper, we introduced HealthLens, a novel and userfriendly visualization tool designed to simplify the process of exploring and analyzing electronic medical records. By leveraging the MedCodeT5 model and a bilevel optimization strategy, HealthLens translates natural language queries into accurate, visually compelling, and clinically relevant visualizations using DVLs. This eliminates the need for users to possess technical expertise, making advanced data visualization accessible to healthcare professionals and researchers alike. Through the demonstration scenario, we showcased how HealthLens empowers users to intuitively interact with medical data, refine visualizations dynamically, and extract actionable insights with minimal effort.