EchoGPT: An Interactive Cardiac Function Assessment Model for Echocardiogram Videos

1 Introduction

In the realm of modern medicine, the continuous development of new materials and the proliferation of wearable devices have led to an increasing demand for portable physiological data monitoring devices. Particularly in the field of echocardiogram video monitoring, advancements in Bioadhesive Ultrasound (BAUS) [Liu et al., 2024b; Wang et al., 2022] technology have opened new possibilities for cardiac health monitoring. However, given that the video data collected by wearable devices tend to be of considerable duration (usually exceeding one hour), it is extremely challenging for physicians to manually diagnose echocardiogram videos, and the accuracy of such diagnoses cannot be guaranteed. Automated diagnosis of echocardiogram videos has now become a research hotspot.

∗Corresponding author.

In order to address the aforementioned issue, it is necessary to develop a multimodal model capable of diagnosing based on echocardiogram videos. Multimodal models like CLIP, including BioMedCLIP, PubMedCLIP and EchoCLIP [Radford et al., 2021; Zhang et al., 2024b; Eslami et al., 2023; Christensen et al., 2024], excel in understanding tasks and offer preliminary diagnostics. However, they often overlook patients' comprehension, hindering effective communication. Additionally, their accuracy in medical metrics and general question-answering capabilities are limited compared to larger language models, restricting their interactive and precise utility. With the development of large language models [OpenAI et al., 2024; Sun et al., 2025], especially multimodal large language models, a new avenue has been paved for patients to gain immediate insights into their health conditions. Existing video-text multimodal large models [Wang et al., 2024; Ataallah et al., 2024a; Zhang et al., 2024c; Cheng et al., 2024], trained mostly on general datasets, struggle to accurately capture the features of medical data, resulting in inaccurate responses. In some medical imaging fields, relevant multimodal image-text large language models (LLMs) have emerged. For instance, SkinGPT [Zhou et al., 2024] is a multimodal LLM designed for the diagnosis of dermatological images. However, there remains a dearth of corresponding multimodal LLMs in the domain of medical video diagnosis, especially echocardiogram video.

In light of these challenges, as shown in Figure 1, this study introduces an innovative medical diagnostic large language model tailored for echocardiogram videos-EchoGPT. This model combines the powerful communication and comprehension capabilities of large language models with the diagnostic capabilities of traditional medical models, enabling patients to receive accurate medical indicator data and understand their health conditions through interactive questioning with the model. This not only enhances patients' selfmonitoring capabilities and alleviates the workload of physicians but also supports local deployment on personal computers, effectively safeguarding users' personal privacy.

The primary contributions of our paper are as follows:

• Pioneering Application of LLM in EchocardiogramDiagnosis: To the best of our knowledge, EchoGPT is the first multimodal echocardiogram diagnosis large language model (LLM). EchoGPT represents the first instance of leveraging LLM in the field of medical video modeling, especially echocardiogram video diagnosis, marking a significant advancement in the application of AI technology for cardiac health monitoring.
• Local Deployment for Enhanced Privacy and Accessibility: EchoGPT's capability for local deployment on personal computers positions it at the forefront of privacy protection in medical diagnostics. As advanced sensors become more ubiquitous, the model's advantages in safeguarding user privacy and facilitating convenient access to healthcare insights will become increasingly evident.
• State-of-the-Art Performance in Diagnostic Accuracy and Interactivity: EchoGPT achieves cutting-edge performance in both diagnostic accuracy and interactivity. The model's sophisticated understanding of echocardiogram videos and its ability to engage in interactive questioning with patients set new standards for AI-assisted medical diagnostics.

2 RelatedWork

Medical Vision-Language Models. The integration of visual and linguistic data in medicine has led to advanced models that boost understanding and analysis. These models handle both visual and textual medical data, aiding in diagnosis, research, and patient care. This development is a major step in medical AI, offering tools to enhance healthcare professionals' expertise and patient outcomes. While CLIP

[Radford et al., 2021] has been influential, its general training has limitations in medical contexts. PubMedCLIP [Eslami et al., 2023], fine-tuned on radiographic data, shows better performance in radiology tasks. BioMedCLIP [Zhang et al., 2024b], pre-trained on scientific data, outperforms general models in biomedical tasks. EchoCLIP [Christensen et al., 2024], tailored for echocardiogram videos, improves performance on relevant datasets. However, these models still face challenges in user interactivity and data accuracy due to CLIP's inherent constraints.

Video-Text Multimodal LLMs. In the general video description domain, multimodal LLMs such as Alibaba's Qwen2-VL [Bai et al., 2023; Wang et al., 2024] with visual enhancement, Peking University's LLava-Video [Zhang et al., 2024c] with two-stage training, and Video-Llama2 [Zhang et al., 2023; Cheng et al., 2024] from the VideoLLaMA series are advancing rapidly. MiniGPT4-Video [Ataallah et al., 2024a; Ataallah et al., 2024b; Chen et al., 2023; Zhu et al., 2023] is notable for translating visual features into LLM space. However, these models perform poorly in medical data comprehension despite excelling in general video understanding and interactivity. These AI models integrate visual and linguistic processing for innovative healthcare applications. In medical 2D imaging, numerous multimodal image-text LLMs have emerged, such as SkinGPT for dermatology, CheXGPT for chest X-rays, Dia-LLaMA for CT report generation, ECG-LLM for electrocardiography, and miniGPT-Med for general medical visual question answering [Zhou et al., 2024; Gu et al., 2024; Chen et al., 2024; Yu et al., 2024; Alkhaldi et al., 2024]. Nevertheless, there are still no relevant multimodal LLMs applied to echocardiogram video diagnosis.

3 Method

3.1 ProblemFormulation

The model framework proposed in this study aims to achieve automated assessment of cardiac function through the integrated analysis of echocardiogram videos and user-submitted textual queries. The inputs to the model include an echocardiogram video x and a user query text y. The format of the video data x is defined as x ∈ RH×W×C×T , where H, W , C, and T represent the height, width, number of color channels, and the number of frames of the video, respectively. For the user query text y, its format is defined as y ∈ RN×M , where N denotes the number of words, and M denotes the vector dimension of each word. The objective of this research is to ultimately output an echocardiogram video xseg with a left ventricular segmentation mask to provide a visual result to aid in clinical diagnosis. Furthermore, we aspire to generate a medical diagnostic report that includes the specific left ventricular ejection fraction (LVEF) and enable the model to provide corresponding responses to user queries y.

3.2 Model Framework of EchoGPT

Our model framework, as depicted in Figure 2, comprises the following three parts:

Part 1: Left Ventricle Segmentation. The objective of this module is to precisely segment the left ventricle region from echocardiogram videos, providing visual reference and guidance for both physicians and users. The input video x is first divided into a sequence of frames, then deep visual features are extracted through an Atrous Convolution network, followed by a linear layer that maps these features to a segmentation mask space. The final output is a video with a left ventricle segmentation mask, offering physicians assisting visual diagnostic results and aiding users in more intuitively understanding the diagnostic outcomes.

Part 2: LVEF Prediction. In this part, the model aims to predict the left ventricular ejection fraction (LVEF) based on echocardiogram videos. Visual features are extracted from the video data x through 3D CNNs and spatiotemporal convolutional networks, followed by a linear layer that maps these features to the ejection fraction space to obtain the predicted LVEF value. This LVEF value serves as prompting information, transmitted alongside the user's query to the Text Tokenizer to obtain corresponding textual features, which are then mapped into the context space of the large language model LLM.

Part 3: Finetuning Video-Text LLM. The inputs for this part include the echocardiogram video x, the user query text y, and the LVEF value obtained from Part 2. The video x is divided into a sequence of frames and visual features are extracted through a Vision Transformer (ViT). To accommodate the context window limitations of the LLM, we perform a concatenate operation on the visual tokens output by ViT to reduce the number of tokens, achieving information compression. The compressed visual features are then mapped to the context space of the LLM through a linear layer. After integrating the aforementioned context features, the LLM generates corresponding diagnostic reports or responses based on the user's query.

3.3 Left Ventricle SegmentationModel

This model employs a semantic segmentation architecture based on DeepLabV3-ResNet50 [Chen et al., 2017], incorporating Atrous Convolution technology, dedicated to the segmentation task of the left ventricle in echocardiogram video sequences. Given that the echocardiogram video sequences in the EchoNet-Dynamic dataset are only annotated with expert segmentation masks at two key frames, end-systole and end-diastole, this research fully leverages this sparsely annotated characteristic and devises corresponding weakly supervised training strategies. The core features of the adopted DeepLabV3-ResNet50 model are reflected in: multi-scale feature extraction, capturing contextual information through Atrous Convolutions with varying dilation rates; spatial pyramid pooling, integrating multi-scale feature representations; and an encoder-decoder structure, ensuring the precision of segmentation boundaries. After processing the input data through Atrous Convolution, informative feature maps are obtained, which are ultimately mapped to binary segmentation masks via a linear layer to distinguish between left ventricular and non-left ventricular regions. In terms of training strategy, this study opts to train the model from scratch,

Algorithm 1: Forward Propagation of EchoGPT

Input : Echocardiogram video x ∈ RH×W×C×T , User question text y ∈ RN×M

Output: Segmented video xseg, LVEF value, Responses

• 1 Part 1: Left Ventricle Segmentation
• 2 for each frame in video x do

3

• extract frame features using Atrous Convolution 4 project features to segmentation mask space using a linear layer

• 5 end
• 6 combine segmentation masks to get xseg
• 7 Part 2: LVEF Prediction
• 8 extract visual features from x using 3D CNN and Spatiotemporal convolutions
• 9 map features to LVEF space using a linear layer to get LVEF
• 10 Part 3: Finetuning Video-Text LLM
• 11 for each frame in video x do
• 12 extract frame features using Vision Transformer 13 end
• 14 concatenate ViT output to reduce token numbers
• 15 map visual features to LLM context space using a linear layer
• 16 generate responses based on y, LVEF, and context using LLM

rather than directly applying pre-trained weights, to better adapt to the unique characteristics of echocardiogram data. After training for 50 epochs, the segmentation model is capable of accurately segmenting the left ventricle from echocardiogram video sequences, with visualization results that can assist physicians in diagnosis and help users gain a deeper understanding of diagnostic outcomes.

3.4 LVEF PredictionModel

This model employs a three-dimensional convolutional neural network architecture based on R2Plus1D-18 [Tran et al., 2014; Tran et al., 2017], aimed at predicting the ejection fraction of echocardiogram videos through spatiotemporal feature extraction. The method fully leverages the advantages of transfer learning from pre-trained models and has been specifically optimized for the characteristics of echocardiogram images. Specifically, the model utilizes the pre-trained R2Plus1D-18 as the backbone network and incorporates the following key improvements to adapt to the ejection fraction prediction task: firstly, replacing the original classification layer with a single-output linear layer for regression prediction; secondly, retaining the feature extraction capability of spatiotemporal convolutional layers to capture key spatiotemporal features in the videos; and lastly, integrating residual connections to ensure the effective propagation of deep features. With these enhancements, the prediction model can effectively predict the corresponding left ventricular ejection fraction from echocardiogram videos. The obtained precise ejection fraction indices will be combined with user queries as prompts, transmitted to the large language model, to achieve more accurate interaction and analysis.

3.5 Finetuning Video-Text Large Language Model

The training process in this part is divided into two stages: Stage1 : Echocardiogram video-Medical Report pair pretraining. In this stage, the model processes multiple video frames to comprehend the video content. Specifically, a maximum of N frames are sampled from each video, with N, equal to 45, determined by the context window size of the language model. We employed predefined prompts with the following template: < s > [INST] < Img >< FrameFeature1 > ... < Img >< FrameFeatureN >< Instruction >< /INST >. A linear layer was trained to project the visual features encoded by the visual encoder [Dosovitskiy et al., 2021; Li et al., 2023; Sun et al., 2023; Chen et al., 2020] (e.g., ViT) into the text space of the LLM [Touvron et al., 2023; Zhang et al., 2024a], utilizing image description loss (captioning loss). The encoded video frames were input into the model alongside the corresponding medical reports; < FrameFeature > in the prompt was replaced with the encoded sampled video frames, and < Instruction > was replaced with randomly selected instructions from a predefined set, which included the left ventricular ejection fraction (LVEF) corresponding to the video, such as "Generate a report according to this video whose LVEF is 56%." The EchoNet-Dynamic dataset, annotated with corresponding medical report data, was used for large-scale medical report generation training.

Stage 2 : Echocardiogram Video question answering instruction finetuning. The same training strategy as the first stage was employed in this stage, but with a focus on using high-quality medical visual question-and-answer datasets for instruction fine-tuning [Liu et al., 2024a]. This helps enhance the model's ability to interpret input videos and generate precise answers to corresponding questions. The template was the same as in the second phase, except that < Instruction > was replaced with general questions mentioned in the question-and-answer dataset. The EchoNet-Dynamic dataset, annotated with relevant medical visual question-and-answer data, was used for medical visual question-and-answer training.

Regarding training details, both stages maintained a batch size of 2 and utilized the AdamW optimizer with a cosine learning rate scheduler, setting the learning rate to 1e-4. The visual backbone was the Vision Transformer with frozen weights. The linear projection layer was trained from scratch, and the LoRA (Low-rank Adaptation) method [Hu et al., 2021] was used for efficient fine-tuning of the large language model Llama2-7B-Chat. Specifically, the Wq and Wv components were fine-tuned, with the rank (r) set to 64 and the LoRA-alpha value equal to 16. The entire model maintained a consistent image resolution of 224×224 pixels throughout all stages to ensure uniformity.

4 Experiments

4.1 Datasets and EvaluationMetrics

Datasets

Table 1 presents a detailed overview of the publicly available echocardiogram video dataset EchoNet-Dynamic, our corresponding diagnostic report annotation dataset, and the associated medical visual question-answering dataset.

EchoNet-Dynamic Dataset. The EchoNet-Dynamic dataset [Ouyang et al., 2020; Ouyang et al., 2019] includes over 10,000 cardiac ultrasound videos from Stanford University Medical Center, featuring diverse patients. Each video, recorded from the apical four-chamber view, comes with clinical data like EF, ESV, and EDV, and includes left ventricular mask annotations by doctors at end-systolic and end-diastolic phases.

Diagnosis Report Annotation of Echocardiogram Videos. We invited an experienced echocardiogram physician to annotate each video with detailed medical reports. These reports describe the videos and assess left ventricular pump function using LVEF, aiming to support cardiac function analysis and clinical decisions.

Construction of Video Question Answering (VQA) Dataset. Our VQA dataset includes three question types: descriptive, explanatory, and advisory. Descriptive questions help users understand echocardiogram video features. Explanatory questions provide insights into cardiac ultrasound data for better heart health comprehension. Advisory questions offer professional advice on treatment and lifestyle changes, particularly for those at risk of heart pump insufficiency or failure, along with early intervention suggestions.

Evaluation Metrics

Left ventricular ejection fraction (LVEF) is a critical cardiac performance metric that quantifies the percentage of blood pumped out of the left ventricle with each contraction, reflecting the heart's ability to efficiently circulate blood. To systematically assess the performance of the model proposed in this study compared to other existing models in predicting the accuracy of LVEF, a series of quantitative metrics were adopted for experimental evaluation:

• AUC (40%): Measures the model's ability to distinguish between LVEF values below and above 40%.
• AUC (50%): Assesses the model's capability to differentiate LVEF values below and above 50%, important for identifying cardiac insufficiency.
• AUC (70%): Evaluates the model's performance in extreme cases.
• R²: Reflects the proportion of LVEF variation explained by the model.
• MAE (Mean Absolute Error): Measures the average difference between predicted and actual LVEF values.

In summary, these metrics collectively form a multidimensional assessment framework for comprehensively evaluating and comparing the performance of different models in predicting LVEF.

4.2 Experimental Implementation

Quantitative Experiment. In the quantitative segment of this study, a comparative experiment was conducted to evaluate the performance of various models in predicting the accuracy of LVEF. Specifically, we selected models such as EchoCLIP, CLIP, BioMedCLIP, and PubMedCLIP [Christensen et al., 2024; Radford et al., 2021; Zhang et al., 2024b; Eslami et al., 2023], and applied them to over 1200 test videos from the EchoNet-Dynamic dataset to predict left ventricular ejection fraction. By comparing the predicted outcomes of these models with actual clinical data, we quantified their accuracy and performed statistical analysis. Concurrently, in order to empirically validate the efficacy of EchoGPT in practical applications, we have solicited feedback from a cohort of seasoned physicians specializing in echocardiogram through a structured questionnaire survey. This methodological approach was employed to gather expert opinions and assess the real-world utility of the model within the domain of cardiac ultrasound imaging.

Qualitative Experiment. In the qualitative experimental phase, we focused on assessing the capabilities of largescale visual-text multimodal language models in generating diagnostic reports and answering medical-related questions. To this end, we selected models including MiniGPT4Video, Qwen2-VL, Video-Llama2, and Llava-Video [Ataallah et al., 2024a; Wang et al., 2024; Cheng et al., 2024; Zhang et al., 2024c], and employed a case study approach to demonstrate the effectiveness of these models in practical applications by presenting specific cases to reflect the differences in their performance. In conjunction with this, to substantiate the efficacy of our two-stage training strategy, we conducted ablation studies to compare the actual performance of EchoGPT, which underwent the complete two-stage training regimen, with that of EchoGPT variants trained solely in stage 1 and stage 2, respectively. This comparative analysis was designed to elucidate the contributions of each training phase to the overall performance of the model.

All experiments were conducted in a high-performance computing environment equipped with four RTX-4090 GPUs and an Intel(R) Xeon(R) Platinum 8336C CPU, ensuring the sufficiency of computational resources and the reliability of the experimental results.

4.3 Quantitative Results

Comparison of Predictive Accuracy for Left Ventricular Ejection Fraction (LVEF). In our study, we compared the performance of various models in predicting left ventricular ejection fraction (LVEF). The experimental results in Table 2 demonstrated that our proposed EchoGPT model outperformed all other models across all evaluated metrics.

Specifically, EchoGPT achieved AUC values of 0.9519, 0.9221, and 0.8475 for the thresholds of 40%, 50%, and 70%, respectively, which are significantly higher than the performance of other models. This indicates that EchoGPT possesses an exceedingly high degree of accuracy in identifying samples with ejection fractions below 40%, below 50%, and above 70%. In terms of the R² score, EchoGPT reached 0.6848, implying that our model accounts for approximately 68.48% of the variability in ejection fraction, the highest proportion among all compared models. Furthermore, EchoGPT's mean absolute error (MAE) was 7.5934, the lowest among all models, which further corroborates the high precision of EchoGPT in predicting left ventricular ejection fraction. In comparison, other models such as EchoCLIP, BioMedCLIP, PubMedCLIP, and CLIP did not perform as well as EchoGPT.

In summary, the EchoGPT model demonstrated superior performance in predicting left ventricular ejection fraction, both in terms of distinguishing the severity of ejection fraction and overall predictive accuracy, significantly outperforming other models under comparison. These results substantiate the potential application of EchoGPT in the field of cardiac function assessment, offering a robust tool for clinical decision-making.

Questionnaire-based Evaluation of EchoGPT. To assess the diagnostic accuracy of EchoGPT and its practical utility to users, we engaged a panel of five physicians specializing in echocardiogram to evaluate the system's performance. For this evaluation, we randomly selected 100 videos from the test set of the EchoNet-Dynamic dataset and employed a questionnaire-based assessment methodology.We provided EchoGPT with the following prompts to elicit responses:

1. Could you describe this echocardiogram video for me?
2. My LVEF is not in the normal range, what should I do for this case?

Based on the responses provided by EchoGPT, physicians were queried on the accuracy and relevance of its diagnosis, the informativeness of its answers, the utility of its suggestions, its capacity to assist in medical diagnosis, its ability to enhance users' understanding of their health conditions, the privacy protection afforded by local deployment, and their willingness to utilize EchoGPT. The results, as depicted in the table, indicate that in the majority of cases, physicians concurred with EchoGPT's responses, often expressing agreement or strong agreement. Figure 3 indicates that EchoGPT demonstrates commendable performance in both the accuracy of its diagnoses and the practicality of its interactive capabilities.

4.4 Qualitative Results

Ablation Study. To ensure that EchoGPT provides accurate diagnostic reports while maintaining a high level of interactivity, we adopted the two-stage training strategy as mentioned in Section 3.5. To validate the substantial benefits of this training strategy in enhancing the model's actual performance, we conducted ablation experiments to confirm the necessity of both training stages. As illustrated in Figure 4, the first image depicts a communication dialogue between a physician and a patient, while the second image shows an interactive dialogue between EchoGPT and a user. It can be observed that EchoGPT, after undergoing the two-stage training, is capable of accurately providing correct diagnostic reports and responding to user-related inquiries with concise language. The third image demonstrates the responses of EchoGPT trained only in the first stage; although it can provide correct diagnostic reports, it fails to respond precisely and concisely to other user queries, often including redundant information. The fourth image displays the performance of EchoGPT trained solely in the second stage, which evidently cannot provide accurate diagnostic reports. In summary, the results of the ablation experiments substantiate the effectiveness and necessity of the two-stage training strategy. EchoGPT, after being trained in both stages, can achieve diagnostic results comparable to those of physicians and can accurately and concisely answer user-related questions.

Comparative Experiment. To verify the superior performance of EchoGPT in terms of interactivity and diagnostic report generation capabilities, we also compared it with other Video-Text Large Language Models (LLMs). As shown in the four lower images of Figure 4, these illustrate the actual conversational effects of four different Video-Text LLMs with users. MiniGPT4-Video and Video-Llama2 were both unable to correctly identify the characteristics of echocardiogram videos, and even failed to provide correct descriptions of the echocardiogram videos. While Qwen2-VL and LlavaVideo could correctly recognize echocardiogram videos and provide some descriptive language, none of the aforementioned models were able to calculate the left ventricular ejection fraction, and thus could not provide actual diagnostic results. In contrast, the second image at the top demonstrates the conversational effect of EchoGPT, which not only possesses the strong interactivity of large language models but also can provide accurate diagnostic results, effectively assisting users in understanding their health conditions.

EchoGPT on mobile devices and further optimize its conversational capabilities.

Acknowledgments

This work is supported by the Science and Technology Project of Liaoning Province (2023JH2/101700363, 2024JH2/102600027), in part by the National Natural Science Foundation of China under Grant 62072073, 62106034, in part by the Fundamental Research Funds for the Central Universities under Grant DUT24ZD124, in part by the Dalian Innovation Fund 2021JJ12GX016, and the Science and Technology Project of Dalian City (2024JJ12GX025, 2023JJ12SN029 and 2023JJ11CG005).

5 Conclusion

This study successfully developed and validated EchoGPT, a large language model for medical diagnosis tailored to echocardiogram videos. EchoGPT not only enhances patients' ability for self-monitoring and alleviates the workload of physicians but also protects users' personal privacy through local deployment. EchoGPT demonstrates exceptional performance in diagnostic accuracy and user interactivity, setting a new standard for AI-assisted medical diagnostics. Employing a two-stage training strategy, EchoGPT is capable of providing accurate diagnostic reports and responding to user inquiries with concise language. Furthermore, EchoGPT outperforms existing models in predicting left ventricular ejection fraction, garnering recognition from clinical physicians. Future work will explore the local deployment of