TimelyMed: AI-Driven Clinical Course Attribution and Temporal Mapping for Psychiatric Medical Records

1 Introduction

Understanding the longitudinal clinical course, which refers to the progression and development of a psychiatric disorder over time, is crucial for effective decision-making in mental health care. Key factors of the clinical course, including symptom onset, episode recurrence, hospitalization history, and treatment response, are crucial for guiding appropriate interventions [Angst and Sellaro, 2000; Emsley et al., 2013; Treuer and Tohen, 2010]. For example, as illustrated in Figure 1, clinicians often need to determine when psychiatric symptoms first appeared, how many times the patient has experienced psychiatric episodes, and when the most recent psychiatric hospitalization occurred from clinical notes. However, manually extracting and organizing course-related information, such as temporal and medical events, from unstructured medical records is a labor-intensive process, often requiring clinicians to reconstruct patient histories through tedious chart reviews or even hand-drawn timelines [Treuer and Tohen, 2010]. This process is not only time-consuming but also prone to human errors and omissions.

Extracting temporal information from medical records presents a challenging task for large language models (LLMs) [Agrawal et al., 2022; Monajatipoor et al., 2024; Ge et al., 2024]. LLMs may struggle to generate structured outputs that meet specific requirements, hindering downstream tasks. Additionally, they are prone to hallucinations. For example, the sentence "She was titrated to 200 mg on 8/25." might be incorrectly interpreted as referring to "August 25, 2002." Moreover, variations in physicians' writing styles can further impede accurate time recognition. For instance, in the sentence "27 start watching CKMH Psychiatry, MDD was diagnosed." the model may fail to recognize that "27" refers to the patient's age. 1

To address these challenges, we propose TimelyMed, 2 an automated system that extracts temporal and medical event information from psychiatric medical records using a locally deployed LLM. Our system is designed to provide clinicians with an accurate and structured representation of a patient's psychiatric history, reducing the cognitive burden and minimizing errors in timeline reconstruction. TimelyMed leverages a predefined taxonomy of psychiatric events and temporal expressions, allowing it to accurately capture critical courses in a patient's history. Through training on annotated data, we optimize the LLM to extract temporal and medical events from clinical texts. After structuring these elements into a timeline visualization, clinicians can gain a comprehensive view of a patient's mental health trajectory with minimal effort. The attribution feature further enhances the system by providing direct links between extracted events and their corresponding text, allowing clinicians to verify and contextualize each piece of extracted information.

1 We used Llama-3.1-8B-Instruct in examples.

2 http://nlg17.csie.ntu.edu.tw:8501

By converting raw medical records into structured event timelines, TimelyMed enables clinicians to obtain a overview of a patient's psychiatric history, which is critical for both diagnosis and treatment planning. For example, knowing the exact number of psychiatric episodes or the timing of the most recent hospitalization allows for better monitoring of disease progression and focusing on precipitating factors and treatment adherence. Moreover, integrating our system into clinical workflows can support timely decision-making by providing rapid, structured insights from unstructured data sources. This capability is particularly valuable in psychiatric settings, where detailed patient histories are essential for understanding the often chronic and episodic nature of disorders such as schizophrenia, bipolar disorder, and major depressive disorder. The ability to quickly retrieve and synthesize these details frommedical records could also improve the accuracy of psychiatric diagnoses and treatment outcomes.

2 Dataset Construction

This study utilized the Integrated Medical Database of National Taiwan University Hospital (NTUH-iMD), approved by the Institutional Review Board of National Taiwan University Hospital (NTUH-201610072RINA). The NTUH-iMD database includes electronic medical records such as admission and discharge notes, and ICD diagnostic codes. This study focused on 500 discharge notes from the psychiatric unit with a principal psychiatric diagnosis (ICD-9CM: 290-319 or ICD-10-CM: F00-F99) between January 1, 2006, and September 30, 2016. All personal identifying information, including names, addresses, and birthdays, was de-identified, and only sections related to the history of the present illness were included. Furthermore, we apply an additional de-identification tool [Wang et al., 2022; Lee et al., 2024] to replace sensitive details, such as residence, school, workplace, and other elements that could potentially disclose aspects of the patient's private life. Although effective, random replacements can impact data completeness and readability. Therefore, after the tool's application, manual review and modification were conducted to ensure both de-identification and the integrity of the data for

subsequent analysis. 3

All medical chart annotations were performed by a boardcertified clinical psychiatrist with 20 years of experience and a researcher specializing in NLP. Annotators were instructed to label each sentence processed by the Sentence Splitter (Stanza toolkit [Qi et al., 2020; Zhang et al., 2021]). Although modern large language models support increasingly long input contexts, allowing most clinical notes to be processed as input, using sentences as the unit of analysis remains a more effective approach. This decision is based on the limited availability of annotated data, computational constraints, and the need for attribution functionality.

Sentences indicating the development or worsening of psychiatric symptoms were categorized as "Symptom/Episode." Those describing treatment remission (fully controlled) or response (slight improvement) were labeled as "Remission/Response," while sentences mentioning admissions to acute psychiatric or rehabilitation wards were classified as "Hospitalization." If a sentence did not contain any of these events, it was categorized as "None". Examples of these medical event categories are provided in Table 1. Notably, a sentence may contain multiple medical events.

Temporal events include "Date", "Age", "Duration", "Ago", and "Persistence". Among them, "Ago" specifies how many years, months, or days before the present admission, while "Persistence" indicates how long a condition continued after a specific event. All temporal events are labeled in their respective standardized formats. A total of 500 discharge summaries were manually annotated and cross-verified by both annotators. In cases of inconsistency, the two reviewers discuss and reach a consensus.

3 TimelyMed System

TimelyMed comprises an offline training phase and an online serving phase (Figure 2). During the offline training phase, a LLM is trained to extract temporal and medical events. In the online serving phase, the system converts clinical notes into course timelines in real time.

3 The examples in our demo system are written by a psychiatric physician in an anonymous and synthesized manner.

Identify the type of medical events and corresponding temporal events from a given sentence in a clinical note.

# Steps

1. Analyze the Sentence: Break down the sentence to identify keywords or phrases that indicate a medical event type.
2. Determine Medical Event: Categorize the event as 'Episode/Symptom',

'Hospitalization', 'Remission/Response', or 'None' based on the context and identified keywords.

3. Identify Temporal Event: Recognize any temporal indicators to classify the temporal event as 'Age', 'Ago', 'Duration', 'Persistence', 'Date' or 'None'.

# Output Format

Provide the output as a JSON object with two fields, "medical event" and "temporal event", indicating the identified types.

{"medical event": "[Medical Event Type]", "temporal event": "[Temporal Event Type]"} <demonstrations>

Notes

- If the sentence does not clearly indicate any medical or temporal event, classify it as 'None' for one or both categories.

- Pay close attention to temporal cues, as they are crucial for accurately identifying the time event type.

- A sentence may contain multiple medical events; separate them with commas.

Figure 3: System Prompt Design for the TimelyMed System

3.1 Offline Training

Based on annotated data, we leverage supervised fine-tuning to train an Events Extractor to extract temporal and medical events from clinical notes. We employ the LoRA [Hu et al., 2022] to fine-tune LLMs. Besides, we design a system prompt that specifies the task description, event categories, output format, demonstrations, and key constraints, as shown in Figure 3.

3.2 Online Serving

The process begins by segmenting the clinical notes into sentences. A well-trained LLM is then employed to extract both temporal and medical event information. Following extraction, a Temporal Alignment step is applied to format the data according to the requirements of the timeline package. 4 Specifically, all time-related information is converted into a standardized date format to ensure consistency. Additionally, if a medical event is identified without an accompanying temporal event, the system searches the preceding sentence for a temporal event. This process continues iteratively until a relevant temporal reference is found, ensuring that every medical event is appropriately anchored in time. Finally, we will include the source sentences as attribution in the timeline to enhance the system's reliability.

4 https://github.com/innerdoc/streamlit-timeline

4 Experiments and Discussion

We conducted an evaluation of three models: LLaMA (Llama-3.1-8B-Instruct) [Touvron et al., 2023] and Mistral (Ministral-8B-Instruct-2410) [Jiang et al., 2023]. The models were assessed using 10-fold crossvalidation to ensure robust performance estimation. Model effectiveness was measured using Precision, Recall, and F1score as evaluation metrics.

Table 2 and Table 3 present the performance results for medical and temporal event extraction, respectively. The results indicate that Mistral achieved the best overall performance after training. In the medical event task, "Remission/Response" exhibited the weakest performance, likely due to the diverse terminology used by different physicians to describe symptom control and alleviation. In the temporal event task, "Persistence" performed poorly, possibly because of its low occurrence in the dataset (approximately 3.5%).

5 Conclusion and FutureWork

TimelyMed provides an efficient and privacy-preserving solution for extracting and organizing psychiatric course timelines frommedical records. By leveraging a locally deployed LLM, the system ensures data security while enhancing clinical workflows through automated event extraction and attribution. This approach reduces the cognitive burden on clinicians, improves decision-making, and supports more accurate psychiatric diagnoses and treatment planning. Future work will focus on refining event classification and expanding the system's applicability across diverse psychiatric conditions.

Acknowledgments

This work was supported by Ministry of Technology and Science, Taiwan , under grants MOST110-2314-B-400-053MY3, National Science and Technology Council, Taiwan, under grants NSTC 112-2634-F-002-005-, and Ministry of Education (MOE), Taiwan, under grants NTU-113L900901.

Contribution Statement

Author Chien-Hung Chen and Author Chi-Shin Wu contributed equally to this work.