Evaluation of Medical Large Language Models: Taxonomy, Review, and Directions

1 Introduction

Large Language Models have shown promise in several medical applications, but their evaluation requires a nuanced approach that goes beyond simple accuracy metrics. Other papers in the literature have already compared and evaluated how different LLM models perform in general and healthcare applications [Chang et al., 2024; Bedi and Liu, 2025]. This paper, in contrast, assumes that a Large Language Model (LLM) is given and we want to evaluate how effectively this model addresses the medical application at hand, which can be clinical diagnosis, aiding physicians in treatment choice, or helping students understand a specific medical topic.

The assessment of LLM results may follow an automatic or a manual, human-driven approach [Tam et al., 2024]. The protocols and metrics used in these approaches differ, but there are at least three well-defined dimensions one wants to evaluate in any LLM application, namely: (i) model accuracy, (ii) robustness, and (iii) fairness and biases. Model accuracy measures how precise a model is on a given application, and the way it is measured is highly dependent on the application. For example, accuracy can be measured by traditional machine learning predictive measures, such as F1-score, in diagnosis problems, or by the ROUGE score in document summarization tasks. Robustness evaluates how the response of the model changes when data distribution changes, noise is added to data or other challenging inputs are presented to the model. Fairness and bias assess whether the model treats different data groups consistently, where groups can be defined by age, race, gender, etc.

The rigorous evaluation of Large Language Models (LLMs) is paramount for their responsible integration into the medical domain. The potential benefits of LLMs are substantial, but their use without thorough assessment carries risks of unreliability, inaccuracy, and harm. Given the dependence evaluation metrics have on different medical applications, this paper proposes a taxonomy of medical applications and then reviews how different model types (discriminative versus generative) used in different applications imply different evaluation metrics. This paper outlines the key considerations that underscore the importance of evaluating LLMs in this critical domain, emphasizing commonly used datasets and metrics most appropriate to different applications. Additionally, we identify the limitations and challenges of applying LLMs in medical tasks, highlighting why meticulous assessment is not optional, but a fundamental requirement for their safe and beneficial adoption.

This paper fills this gap by introducing a taxonomy of medical tasks, explicitly linking them to LLM evaluation metrics. Unlike previous works that either broadly cover healthcare applications or focus on specific tasks, this study provides a structured approach to medical applications with a strong LLM evaluation perspective. Figure 1 details the structure of this paper. We summarize our contributions to the Medical LLMs evaluation as:

• A comprehensive survey of the relevant literature.
• A novel taxonomy for the evaluation of medical LLMs in different applications.
• A discussion on promising lines for future research addressing current challenges.

• A publically-available repository of metrics, datasets, and an annotated taxonomy (see https://github.com/ MALL-DCC-UFMG/medical-llms-evaluation).

2 Preliminaries

The type of model used to address the medical application, i.e., whether it follows a discriminative or generative model, provides a useful framework for understanding how LLMs process information and the kind of output they provide. It is also key to the type of evaluation that should be performed.

Discriminative Models Discriminative models focus on understanding and categorizing existing data. These tasks are about making distinctions, classifying information, and extracting relevant details from input. In a medical context, this often means analyzing patient records [Karabacak and Margetis, 2023], or medical literature to answer specific questions [Chen and Li, 2023], or categorize information. The goal is to find patterns, relationships, and categories within the given input, and provide structured outputs such as labels, classifications, or extracted data points. For instance, a discriminative task might involve identifying all instances of "diabetes" mentioned in a patient's medical history or classifying a clinical note as either a "diagnosis" or "treatment" record.

Generative Models Generative models, conversely, involve generating text or images based on the given input. These tasks are not about categorizing or extracting information, but rather about producing content that is supposed to be relevant and coherent. In a medical context, generative models might include summarizing a lengthy medical report, generating a simplified explanation of a complex medical condition for a patient [Gao, 2023], or translating medical information into another language [Genovese et al., 2024].

The emphasis is on the model's ability to synthesize new text that adheres to specific instructions and accurately conveys the necessary information.

Evaluation Discriminate and generative models require different approaches and evaluation metrics. Although both will be evaluated according to accuracy, robustness, and fairness and biases, their metrics differ. Discriminative models are often evaluated based on metrics such as accuracy, precision, and recall, which measure how well the model identifies and categorizes information. On the other hand, generative models are evaluated based on the quality of the generated text using metrics such as ROUGE scores, which assess the similarity between the generated text and reference summaries or human-created content.

Besides the three aforementioned evaluation dimensions, generative model evaluation should also consider other dimensions related to text generation, such as factuality and metrics referring to text quality. Factuality evaluates how information or answers provided by the model are consistent with real-world knowledge and is paramount for model trust. It is a way to detect model hallucinations.

3 LLMs Medical Applications Taxonomy

To better understand the capabilities and limitations of LLMs in medicine, we categorize their applications into four classes: (i) clinical decision support and diagnosis (CDSD) (§3.1), (ii) administrative and documentation tasks (ADT) (§3.2), (iii) medical education and research (MER) (§3.3), (iv) miscellaneous applications (§3.4).

3.1 Clinical Decision Support and Diagnosis

Clinical decision support and diagnosis (CDSD) encompass the use of LLMs to aid medical professionals in making informed decisions about patient care [Goh et al., 2024]. We further categorize the decision and diagnosis-based clinical data according to their specific purpose, each one with different evaluation needs.

Medical Question Answering (MQA). This application involves LLMs providing precise answers to clinical questions based on medical knowledge, literature, or patient data. MQA can assist clinicians in quickly finding information about symptoms, treatment options, and drug interactions [Singhal and others, 2023]. Specifically, the task can be open-domain, in which the model answers without any specific reference text being provided, or closed-domain, in which answers are inferred from a supporting text, such as a research abstract. MQA evaluation often involves both automatic - metric usage to assess the accuracy of answers, and manual methods - real clinicians assessing the quality of responses. Multiple datasets are used to evaluate MQA models, such as MedQA [Jin et al., 2021], PubMedQA [Jin, 2019], and MedMCQA [Pal et al., 2022]. These datasets were organized into one of the few benchmarks for medical LLMs, namely MultiMedQA [Singhal and others, 2023]. HealthSearchQA [Singhal and others, 2023], which focuses on realworld health queries, is also a popular dataset choice.

Diagnosis. LLMs can analyze patient data to suggest potential diagnoses. Diagnosis model evaluations have primarily been conducted in simplified medical settings, such as standardized hypothetical patient scenarios or structured clinical case assessments [Eriksen et al., 2024; Kanjee, 2023]. In both instances, all necessary diagnostic information is presented upfront, and the task involves selecting a single correct answer from a predefined list of options. However, LLMs deployed in a high-stakes clinical setting must not only demonstrate exceptional accuracy but also comply with diagnostic and treatment guidelines, exhibit robustness, and effectively follow instructions. These criteria have yet to be thoroughly evaluated in prior medical assessments. Tackling these limitations, in [Rao, 2023] the authors evaluated an LLM across the diagnostic clinical workflow by using curated lists of potential answers and analyzing hypothetical clinical vignettes. Since it is unrealistic to assume that all necessary information is readily available in real-world clinical settings, efforts are being made to assess LLMs' ability to autonomously gather information and perform open-ended diagnoses [Hager et al., 2024].

Treatment Planning. LLMs can support treatment planning by suggesting personalized recommendations based on the latest clinical evidence and patient-specific factors. They have also been used to predict patient care trajectories and identify potential treatment complications [Andrew, 2024]. For instance, in [Pagano, 2023], the authors propose the usage of LLMs in the orthopaedics recommendation treatment based on medical records from patients presenting hip or knee disorders. They measured agreement between the model and specialists through Cohen's Kappa coefficient. Furthermore, the evaluation of LLM-based treatment planning may also be dependent on the medical speciality. For instance, in mental health, certain evaluation criteria hold greater significance than others. A structured assessment framework for clinical LLMs - placing primary emphasis on risk and safety, followed by feasibility, acceptability, and effectiveness - is consistent with established guidelines for evaluating digital mental health applications [Stade and others, 2024].

Clinical Reasoning. This task uses LLMs to simulate a diagnostic process, providing interpretable predictions or diagnoses. Traditional evaluation settings for clinical reasoning focused on single-turn evaluations, in which the model is provided with complete clinical information and asked to select the correct answer [Jin et al., 2021]. However, this evaluation setting fails to measure proactive reasoning, such as asking follow-up questions when necessary. To better simulate realworld scenarios multi-turn evaluations were introduced. The authors in [Li and others, 2024] propose a Patient System - simulating incomplete medical records, and an Expert System - an LLM that must seek additional information before diagnosis. They evaluated the Patient system with factuality and relevance metrics while investigating LLM accuracy performance for varying levels of input information.

Medical Imaging Analysis (MIA). LLMs have been used to analyze several types of medical images, including Xrays, CT scans, and MRIs to assist in diagnosis and treatments. This includes discriminative tasks such as image classification, segmentation, and object detection. MIA tasks are often used in diagnostic and treatment workflows. For example, in the evaluation of LLM-based radiology report generation models, the MIMIC-III [Johnson, 2016] and MIMIC-IV [Johnson et al., 2023] datasets are predominantly used for training and assessment, as they represent the largest publicly accessible free-text electronic health records (EHRs). Standard automatic evaluation metrics encompass lexical approaches, including BLEU [Papineni et al., 2002], ROUGE [Lin, 2004], and METEOR [Banerjee and Lavie, 2005], alongside semantic-based methods such as BERTScore [Zhang et al., 2019]. To enhance domainspecific assessment, methods to provide automatic labeling of exams such as CheXbert [Smit, 2020] and RadGraph [Jain and et. al., 2021] have been introduced. Other works, such as RadCliQ [Yu, 2023], propose application-driven metrics adapted from standard accuracy metrics, offering a more nuanced evaluation of report accuracy and clinical relevance within radiology.

Pathology-related Applications. Pathology data refers to structured and unstructured medical data derived from tissue samples examination, cells, and genetic information to diagnose diseases, assess disease progression, and guide treatment decisions. Given a pathology image, the task consists of answering questions about the clinical findings contained in the image, i.e., a visual question answering task [Delbrouck and Chambon, 2022]. Standard metrics for evaluating LLMgenerated responses include accuracy [Malinowski and Fritz, 2014] and BLEU [Papineni et al., 2002]. While these metrics assess agreement with a gold standard, they do not adequately capture the factual accuracy of generated outputs, which remains a critical challenge[Zhang, 2024]. To address this gap, recent research has introduced domain-specific metrics, such as the F1-RadGraph score [Delbrouck and Chambon, 2022], which evaluates the factual correctness and completeness of

Applications	Dataset	Source	Granularity	Modality	MLModel
	MIMIC-III [Johnson, 2016]	Patients admitted to ICU at BID Medical Center in Boston.	53,423 hospital admissions with 38,597 unique pa- tients.	T I A	D G
CDSD	MIMIC-IV [Johnson et al., 2023]	Patients admitted to ICU at BID Medical Center in Boston.	431,231 hospital admissions with 180,733 unique patients, and 73,181 ICU admissions with 50,920 unique patients.	T I A	D G
	MIMI-CDM [Hager et al., 2024]	Created using the MIMIC-IV database.	2,400 unique patients presented to the emergency de- partment with acute abdominal pain.	T I A	D G
	PathVQA [He et al., 2020]	Images and captions extracted from online digital libraries.	Over 30,000 questions for medical visual question answering.	I Q	D
	SLAKE [Liu and et. al., 2021]	Curated using open-source medical datasets.	642 annotated images referring to 12 diseases and 39 organs.	I Q	D G
ADT	MEDALIGN [Fleming and et. al., ]	Curated from real instruction collection provided by clinicians.	983 natural language instructions for EHR data, cu- rated by clinicians from 7 specialities.	T	G
	ProbSum [Gao, 2023]	Extracted from the MIMIC-III dataset.	Progress notes and unlabeled numerical data (e.g test results).	T	G
	ICD billing codes [Soroush, 2024]	Disease and procedural codes extracted from EHRs.	23,647 disease and 3,673 procedural codes.	T	D
	ACI-Bench [Yim et al., 2023]	Medical notes were created using a seq2seq model.	207 doctor-patient conversations and patient visit notes.	T	G
	MTS-Dialog [Abacha, 2023]	Conversations created using clinical note sections.	1,700 doctor-patient conversations (16k turns, 18k sentences) and summarized clinical notes (6k sen- tences).	T	G
	PriMock57 [Papadopou- los Korfiatis et al., 2022]	Mock consultation recordings with 7 clinicians and 57 actors posing as patients.	57 simulated primary care visits, including record- ings, transcripts, and notes.	T S	G
MER	MedQA-USMLE [Kung and others, 2023]	Derived from the US Medical Licensing Exam- ination (USMLE) containing medical cases and queries.	376 Publicly-available tests.	T	D
	Medical Flash Cards [Han et al., 2023]	Rephrased Q&A pairs from the front/back of med- ical flashcards.	33,955 Q&A pairs.	T	G
	PubMedQA [Jin, 2019]	PubMed abstracts and curated research questions.	211,300 generated Q&A instances.	T	D
	NCBI Disease [Do˘gan et al., 2014]	Manual curation.	793 PubMed abstracts annotated, 6,892 disease men- tions, 790 uniq. disease concepts.	T	D
	S2ORC [Lo et al., 2020]	English academic papers.	4.9M papers with 75B+ tokens.	T	D
Other	ORBIT-Surgical [Yu et al., 2024]	Robotic surgical tasks.	14 simulation tasks.	P	D
	SMM4H [Weissenbacher and others, 2019]	Sourced from Twitter	19,699 Twitter annotated posts.	T	D
	COMETA [Basaldella, 2020]	Curated from health-themed subreddits.	20k biomedical entity mentions with links to SNOMED-CT medical knowledge graph.	T	D

MER - Medical Education and Research. Data modalities: T Text, I Image, A Tabular, S Speech, Q Question-answering pairs, P

Physics-based simulation data. Machine Learning model: D Discriminative, G Generative.

the generated clinical text.

Monitoring. LLMs have been used to monitor patients in real-time, analyzing data from wearable devices and other sources to identify health issues [Pilowsky, 2024]. This can include monitoring vital signs, symptoms, and other healthrelated data to help improve accuracy and reliability in realtime patient care. However, there exist few medical LLMs that can process physiological time-series data, such as electrocardiograms (ECGs) [Li, 2024] and sphygmomanometers (PPGs) [Liu and et. al., 2024].In [Li, 2024], the authors propose a multi-modal self-supervised learning model that uses electrocardiograms (ECGs) and automatically generates clinical reports. They evaluate the model with PTB-XL - for training and testing, and the MIT-BIH - for external validation. Given the discriminative nature of their task (i.e. classification), they used accuracy and F1-score for evaluation. In [Liu and et. al., 2024], the authors investigate the use of LLMs for estimating blood pressure based on physiological features from electrocardiogram (ECG) and photoplethysmogram (PPG) signals using an instruction-tuned approach.

As validation, they used the CAS-BP dataset and regressionrelated metrics.

3.2 Administrative and Documentation Tasks

The evaluation of LLMs for administrative and documentation tasks is essential due to the direct impact they have on clinical workflows, physician well-being, and patient safety. It has the potential to alleviate the administrative burden on clinicians, improve the accuracy of medical records, and enhance healthcare efficiency, all while mitigating the risk of errors and potential harm. The main challenges of these tasks are ensuring that LLMs capture essential clinical details while handling contextual understanding and interpreting nuanced medical language and terminology effectively.

Clinical Documentation. Generating clinical documentation is a time-consuming task, and LLMs can assist professionals in drafting medical documents, streamlining research processes, and improving the efficiency of medical workflows. They can also help with report generation, such as radiology reports or discharge summaries. However, challenges persist, such as accuracy, maintaining the integrity of biomedical research, and addressing the potential for errors. Additionally, the authors in [Van Veen and others, 2023] show that LLM-generated reports tend to be less concise compared to human-written ones.

Electronic Health Record(EHR) Management. Effective management of EHR is essential for organizing and managing patient data. LLMs can automatically identify and categorize critical information, such as symptoms, medications, diseases, and lab results, from patient EHRs. However, the complexity of medical language and the diversity of medical contexts can make it difficult for LLMs to accurately capture the nuances of clinical practice, and they may be prone to errors in interpretation [Karabacak and Margetis, 2023].

Clinical Coding. Clinical coding, such as the International Classification of Diseases (ICD), is a key task for standardizing diagnostics, procedures, treatments, medical billing and reimbursement. LLMs can assist in clinical coding and formatting to improve efficiency and accuracy in this process. However, there are challenges in implementing LLMs for clinical coding, including the specialized nature of the content, and the need to ensure that they are up to date with the latest standards. Clinical coding, which is frequently framed as a multi-label classification task, is often developed and evaluated using the MIMIC dataset. The performance of models is measured using metrics such as F1 score, Area Under the Curve (AUC), and precision@k, which take into account either the top k most frequent labels or the entire label set [Wang, 2024].

Summarization. Summarization of medical texts is an important task that can reduce the burden of analyzing large volumes of clinical information. LLMs have shown promise in medical text summarization, including radiology reports, progress notes, and doctor-patient dialogue [Gao, 2023]. However, there are challenges in evaluating the quality of summaries and ensuring that they are complete, correct, and concise. Furthermore, current automatic evaluation methods for clinical report generation primarily focus on lexical metrics, which can lead to biased and inaccurate assessment of the contextual information present in the reports.

Medical Language Translation. Medical language translation can be divided into two key areas: translating medical terminology between languages and adapting medical dialogue for easier interpretation by non-professionals [HerreraEspejel and Rach, 2023; Genovese et al., 2024]. Both are essential for effective communication across different groups. Ensuring accuracy, preserving clinical meaning, and maintaining consistency across languages are critical. Human evaluation by bilingual medical experts is necessary to validate nuanced medical concepts, while comprehension tests with laypersons help assess the effectiveness of simplifying medical jargon for patients.

3.3 Medical Education and Research

The evaluation of LLMs in Medical Education and Research is essential, due to the unique challenges within these domains, requiring a high degree of accuracy, reliability, and ethical considerations [Preiksaitis and Rose, 2023]. It is necessary to ensure that LLMs enhance, rather than hinder, learning and discovery in medical contexts, while also addressing the specific demands of these tasks. In medical education, the primary challenges include the risk of diminished critical thinking and reasoning skills in students, the potential for misinformation and bias, a lack of emotional understanding in LLM interactions, and the importance of balancing personalized learning with human guidance.

Medical Education. The use of LLMs in medical education is a rapidly growing area with the potential to enhance learning and teaching. LLMs can provide summaries, presentations, translations, explanations, and step-by-step guides on many topics, with customizable depth and style [Preiksaitis and Rose, 2023]. They can also be used to create interactive learning simulations, for instance allowing students to practice conversations with virtual patients [Preiksaitis and Rose, 2023]. Here, ensuring the accuracy of the information, mitigating potential biases, and preventing over-reliance on the output of LLMs continue to be challenges.

Scientific Research. LLMs can significantly enhance the efficiency and effectiveness of scientific research in the medical domain by analyzing large scientific datasets, generating research hypotheses, and summarizing complex research findings [Cascella and et. al., 2023]. However, there is a need for experimental validation to ensure scientific accuracy and the risk of plagiarism or copyright issues [Mishra, 2024]. It is also important to address issues of transparency and explainability to ensure the integrity of medical research.

Literature Review. LLMs can be useful tools for literature review, aiding in the summarization of vast amounts of information into a concise, readable format. They can assist in identifying relevant papers, extracting key findings, and generating comprehensive reviews of specific topics [Chen and Li, 2023]. Despite these potential benefits, it is important to acknowledge challenges such as the possibility of fabricated information and the lack of sophisticated evaluation methods that can account for the nuances of clinical information.

Medication Management. Medication management is an area where LLMs can support patients by providing information on medication dosages, side effects, and interactions, as well as helping them adhere to their medication schedules. These systems can also monitor drug safety and provide alerts about potential adverse effects. A crucial challenge here is the need for LLMs to provide up-to-date information and advice in a clear, concise manner, avoiding the generation of any information that might be inaccurate, incomplete, or misleading.

3.4 Miscellaneous Applications

While some medical applications, such as Q&A, have been assessed using metrics like accuracy [Kanjee, 2023; Samaan, 2023], many applications lack well-defined evaluation methods that capture the nuances of human judgment and the complexity of real-world scenarios. For example, LLMs have been used in medical robotics and for medical product safety monitoring, but the means of evaluating their performance is not always clear. Moreover, the use of LLMs for analyzing social media data introduces further evaluation difficulties, considering the subjective nature of the content.

Medical Robotics. The integration of LLMs with medical robotics has the potential to transform surgical procedures and patient care. LLMs can provide real-time surgical navigation information and analyze physiological parameters, offering intraoperative support to surgeons [Cheng and Sun, 2023]. This capability is especially important in complex procedures, potentially enhancing surgical precision and efficiency. A primary challenge lies in ensuring the accuracy and reliability of LLM outputs, as errors in a surgical context can have serious consequences. Further, the real-time processing demands of surgical robotics require LLMs to have low latency and robust performance in dynamic environments.

Medical Product Safety Monitoring. LLMs can analyze vast amounts of data from various sources, such as clinical trials, patient reports, and social media to monitor the safety of medical products. This is especially relevant in the post-market surveillance phase where real-world data becomes available [Raval, 2021]. Identifying adverse events, drug interactions, and potential safety risks are key advantages of LLMs in this space. The challenge is ensuring that LLMs can distinguish between actual safety signals and background noise in the data, as well as address bias' issues in the datasets. Furthermore, the need to adhere to regulatory requirements and ethical guidelines in data handling adds complexity to the application of LLMs for medical product safety monitoring.

Analysis of Social Media Data. LLMs can be used to analyze social medical data, offering insights into public health trends, patient experiences, and emerging health concerns. This capability allows for real-time monitoring of healthrelated discussions, which can be valuable for identifying potential outbreaks, understanding public perception of health interventions, and assessing patient needs. The use of social media data brings with it many challenges, including the need to filter out misinformation, handle inherent biases in social media data, and ensure that patient privacy is not compromised. Moreover, interpreting the informal language and diverse expressions on social media requires sophisticated natural language processing techniques.

4 Datasets and Benchmarks

When analyzing medical datasets (and benchmarks, defined as sets of datasets used for model evaluation) for evaluating LLMs, several important dimensions need to be considered to ensure the models have a set of desired properties. Here, we present the datasets following the taxonomy defined for medical applications and also categorize them according to the data source, granularity, modality, and type of modeling.

By understanding the medical application (e.g., diagnosis, medical education), the data source (e.g. EHRs, medical literature), and the type of machine learning model (i.e., discriminative vs generative) being addressed, we gain insights into the specific skills an LLM is being used and tested for. Among other possibilities, these skills include knowledge recall, reasoning, text generation, and image analysis. Additionally, categorizing by data granularity (e.g., # patients) and modality (e.g., text, images) allow nuanced evaluation, revealing how LLMs perform across different granularity of medical data and with various types of input. Table 1 shows 19 datasets (and/or benchmarks) widely used by researchers in the medical domain and that have already been processed by LLMs in the four domains of medical application previously defined 1 .

Among the 19 datasets, most focus on textual data, with exceptions being those that use the MIMIC and its derivations and PriMock57, which has audio recordings. For ADT, most works follow a generative approach, as these applications are usually interested in generating patient notes, documents or summaries [Abacha, 2023]. For medical education, most datasets are made of Q&A pairs, and most applications follow a discriminative model -whether the answer to the question is correct or not [Preiksaitis and Rose, 2023]. We also find datasets from social media, where discriminative tasks such as [Weissenbacher and others, 2019; Basaldella, 2020] have been previously addressed. For clinical decision-making, data from MIMIC and its derivations are those with more modalities explored with both discriminative and generative approaches [Johnson, 2016; Johnson et al., 2023; He et al., 2020].

5 EvaluationMeasures

The proper choice of measures/metrics ensures that LLMs' performance is assessed accurately and in alignment with specific medical application requirements. As previously shown, a significant portion of evaluations found in the literature rely solely on accuracy as the primary metric [Samaan, 2023; Kanjee, 2023], especially in tasks involving medical knowledge assessment. This includes multiple-choice Q&A formats, often mirroring medical licensing exams. However, LLM evaluation in medicine is moving toward a more comprehensive, multi-faceted approach that combines traditional metrics with qualitative assessments and addresses issues such as bias and safety. There is an agreement among most researchers that evaluations must move beyond simple benchmarks and examine the real-world clinical impact of these models [Hager et al., 2024].

1 This is not an exhaustive list. Please, find an extended list of datasets in our repository.

Concerning the three dimensions we define as important for the evaluation of LLMs, only accuracy is accounted for in most studies. Regarding robustness, we did not find any study directly evaluating it automatically. This is done mainly by human evaluators [Tam et al., 2024]. Regarding fairness and biases, although there are metrics that can be adapted to consider these dimensions in medical data, specially in discriminative tasks, they are not used. Among these metrics we mention demographic parity difference (DPD), which measures if model predictions vary across different population groups, and equalized odds differences (EOD), which guarantees the model presents equal error rates across different populations [Chang et al., 2024].

Table 2 provides a comprehensive catalog of these standard evaluation criteria currently used, which serves as a valuable reference. The metrics are defined only for generative models, because discriminative models follow traditional metrics, such as accuracy and f-measure [Chang et al., 2024]. For generative models, we classify metrics according to which way they evaluate text quality, i.e., considering syntactic or semantic features, and whether an evaluation of factuality - one of the paramount dimensions - is evaluated. Further, measures for semantic quality are currently applied on top of purely syntactic ones. Also observe that domain-specific metrics are difficult to avoid in certain applications, as they may be the only way to automatically evaluate these methods.

6 Future Directions

Knowing the evaluation of LLMs in medical applications is still far from established, we discuss future research directions to improve evaluation metrics and methodologies.

Clinical Impact. To effectively evaluate medical LLMs, we must prioritize their real-world clinical impact over mere accuracy. This means conducting randomized controlled trials (RCTs) to assess effects on patient outcomes and efficiency, supplemented by observational studies in diverse settings. We need to develop fair and equitable evaluation frameworks that align with patient values and specific clinical uses.

Real-world patient data. Current LLM evaluations often rely on medical exam questions, which don't reflect complex real-world patient data and interactions. Even studies using specific datasets like MIMIC, where a large majority of patients are White, face generalizability challenges across diverse populations and clinical scenarios.

Multilingual datasets. Currently, medical LLMs are predominantly trained on English and Chinese data. To ensure these models accurately understand domain-specific nuances and to promote equitable access to AI's benefits globally, expanding multilingual medical datasets is crucial.

Causal Evaluation. Medical data's complexity often leads to incorrect causal conclusions from observational studies, due to confounding and selection bias. Thus, causal inference is critical to accurately determine the true effects of medical interventions.

Acknowledgments

This work is partially supported by CNPq, CAPES, and FAPEMIG, AWS, and by the projects CIIA-Sa´ude, IAIAINCT on AI.