Abstract
Objective
The use of electronic health records (EHRs) for clinical risk prediction is on the rise. However, in many practical settings, the limited availability of task-specific EHR data can restrict the application of standard machine learning pipelines. In this study, we investigate the potential of leveraging language models (LMs) as a means to incorporate supplementary domain knowledge for improving the performance of various EHR-based risk prediction tasks.
Methods
We propose two novel LM-based methods, namely “LLaMA2-EHR” and “Sent-e-Med.” Our focus is on utilizing the textual descriptions within structured EHRs to make risk predictions about future diagnoses. We conduct a comprehensive comparison with previous approaches across various data types and sizes.
Results
Experiments across 6 different methods and 3 separate risk prediction tasks reveal that employing LMs to represent structured EHRs, such as diagnostic histories, results in significant performance improvements when evaluated using standard metrics such as area under the receiver operating characteristic (ROC) curve and precision-recall (PR) curve. Additionally, they offer benefits such as few-shot learning, the ability to handle previously unseen medical concepts, and adaptability to various medical vocabularies. However, it is noteworthy that outcomes may exhibit sensitivity to a specific prompt.
Conclusion
LMs encompass extensive embedded knowledge, making them valuable for the analysis of EHRs in the context of risk prediction. Nevertheless, it is important to exercise caution in their application, as ongoing safety concerns related to LMs persist and require continuous consideration.
Keywords: electronic health records, risk prediction, large language models, substance use disorder, opioid use disorder
Introduction
Electronic health records (EHRs) are comprehensive digital repositories containing a diverse range of healthcare information, such as patient diagnoses, prescriptions, medications, and laboratory test results1 (as depicted in Figure 1). The utilization of EHRs for various purposes, such as clinical risk prediction,2 has gained increasing attention. The accessibility of past medical information enables the prediction of future diagnoses, hospitalization, readmission, and a variety of associated downstream outcomes.
Figure 1.
Representation of medical records for a single patient in a typical EHR: A visit may have a varying number of medical entities (ie, diagnosis, procedure, medications, etc.).
A frequently proposed real-world use case of EHRs is to stratify specific subpopulations of interest in terms of their risks of future conditions; for example, stakeholders may wish to use local data to identify individuals at high risk of substance use disorder (SUD) or diabetes in order to target appropriate interventions. Unfortunately, in many cases, such data is limited, making it difficult to apply conventional machine learning pipelines. Is it possible to produce high-quality risk scores on a subpopulation of interest by incorporating appropriate prior knowledge?
Previous studies have explored various methods for incorporating prior knowledge into EHRs. For instance, GRAM3 and GBERT4 utilized the hierarchical information within the International Classification of Diseases (ICD)5 ontology to enhance their predictive capabilities. Likewise, MedBERT,6 RareBERT,7 and other similar approaches8,9 employed a transfer learning strategy, utilizing a modified BERT10 architecture for EHRs.
Given the differences in vocabularies and data structure between structured EHR data and unstructured textual data, existing LMs designed for structured EHRs have typically been pretrained from scratch.6,8 However, because of the limited availability of EHR data, models exclusively pretrained on EHRs may not encompass the same breadth of knowledge as models pretrained on larger text corpora. For instance, one key motivation for this study was to identify individuals at a high risk of developing opioid use disorder (OUD) or substance use disorder (SUD), a concern that is becoming increasingly prevalent, particularly in the United States.11,12 There exists a substantial body of literature discussing the associated risks of these conditions,13,14 and having a model equipped with even a portion of this prior knowledge would offer significant advantages.
Historically, clinical notes have typically been rich sources of textual information, and the application of LMs in the context of EHRs has predominantly focused on the analysis of clinical notes. This has given rise to models such as ClinicalBERT15 and GatorTron16, tailored to extract valuable insights from clinical narratives.
In this study, our objective is to explore the feasibility of representing structured EHR data, such as diagnostic history, in the form of textual information. This approach offers greater versatility and independence from specific vocabularies, providing an opportunity to harness the existing knowledge embedded within LMs, among other benefits. The goal is then to investigate how this approach can enhance performance in various risk prediction tasks.
We propose two novel methods based on LMs—“Sent-e-Med” and “LLaMA2-EHR.” Standard risk prediction tasks often revolve around a classification objective where we train a task-specific loss function on a customized dataset. Considering that most large language models (LLMs)17–19 are originally designed for text generation, their application in risk score prediction might seem counterintuitive. We nevertheless show that, when evaluated using standard metrics like receiver operating characteristic (ROC) AUC and precision-recall (PR) AUC, these models significantly outperform more conventional machine learning pipelines. However, we also demonstrate a particular kind of brittleness with respect to changes in the risk-prediction prompt, highlighting the need for extreme caution in moving forward with pipelines based on LLM predictions. We especially call for careful evaluation practices using meaningful validation sets and traditional evaluation metrics.
Materials and methods
Problem statement
For simplicity, we consider the medical record of a single patient. A patient can have multiple visits in their medical history. Let V = [V1, V2, V3, …, VT] represent the list of all visits in chronological order. Our objective is to investigate whether past medical diagnoses can be informative in predicting the future risk of different conditions such as SUD, OUD, and Diabetes. Let Vt = {D1, D2, …, DN}, where Dn is the nth diagnosis of the patient within visit Vt. Note that the number of diagnoses within each visit can vary; for instance, V1 = {D1, D2} and V2 = {D1, D2, D3}. Furthermore, both the number of visits and the intervals between visits can also differ among patients.
Data
We experiment using 3 datasets with differing sources and sizes. Our focus is on utilizing historical diagnoses information derived from patients’ medical records.
SIPPS: This is a de-identified dataset obtained from the Staten Island Patient Provider Service (SIPPS) network. It includes inpatient, emergency room (ER) visits, and outpatient records from 2 different hospitals (ie, Staten Island University Hospital and Richmond Medical Center) representing the population of Staten Island in New York. It contains records from the years 2019–2021. Diagnoses in SIPPS are represented by ICD 10 codes. Due to privacy and ethical considerations, the SIPPS data are not openly shared. However, qualified researchers wishing to access the data for legitimate research purposes may contact the corresponding author for data-sharing inquiries.
MIMIC-IV 20 : This publicly available data is sourced from 2 in-hospital database systems: a custom hospital-wide EHR and an ICU-specific clinical information system. It contains de-identified medical records of patients admitted to the ER or intensive care units at Beth Israel Deaconess Medical Center (BIDMC) from 2008 to 2019. Diagnoses in MIMIC-IV are coded using both ICD 9 and ICD 10 codes.
Synthea 21 : This is an open-source package that generates realistic synthetic patients with associated health records for the top 10 reasons patients visit their primary care physicians and the top 10 chronic conditions responsible for years of life lost. We generated synthetic EHR data using Synthea and used it for different validation experiments. Diagnoses in Synthea are represented by SNOMED CT22 codes which were later mapped to ICD 10 codes for easy analysis.
Data processing
For each risk prediction task, we create 2 groups of patients based on their diagnosis history, as outlined in Figure 2.
Figure 2.
Process of creating patient groups. Patients who had at least one OUD/SUD/Diabetes diagnosis are put into Case Group while those who did not have any OUD/SUD/Diabetes diagnosis are put into Control Group.
Case group: The case group comprises patients who have at least one diagnosis that aligns with the prediction task. For instance, in the context of predicting SUD diagnoses, it includes any patient who has received at least one SUD diagnosis during their second visit or subsequent visits. Once a patient is selected, the target for the next visit prediction is the first visit in which the SUD diagnosis is documented.
Control group: The control group comprises patients who do not have any diagnosis corresponding to the prediction task. For instance, when predicting SUD diagnoses, it includes patients with no SUD diagnosis documented in any of their visits. Once a patient is chosen, the target for the next visit prediction is the last visit in their medical history.
In both groups, all visits up to and including the target visit are kept while the latter visits are discarded.
Selecting data for pretraining and finetuning
One major goal of this study is to explore different strategies for using LMs in EHR analysis, which includes the option of either initializing the LM from scratch or fine-tuning an already existing LLM. To facilitate this, we partition the dataset into pretraining and fine-tuning sets.
For pretraining, we leverage the MIMIC-IV dataset, which has been widely used in various studies.3,23,24 Our objective is to explore the potential of this publicly available dataset in learning generalizable representations that can be applied to downstream tasks. The dataset is diverse, encompassing over 114K patient records, and is not specific to any particular disease. The data are partitioned into training, test, and validation sets in a ratio of 7:2:1, respectively. To prevent data leakage, we ensure that the records of each patient are exclusively included in one of the sets. Consequently, the pretraining set consists of approximately 84K patients. The validation and test sets are not utilized during pretraining and are reserved for evaluation during finetuning.
We utilize 4 different types of data for finetuning:
MIMIC: Obtained from the original MIMIC cohort but is condition-specific (ie, separate data for SUD, OUD, and Diabetes risk prediction).
MIMIC-small: Several smaller samples were chosen from the MIMIC dataset to explore how different models perform with limited data.
SIPPS: As above-mentioned, this is a private EHR data sourced from members within the SIPPS network.
Synthea-small: As with MIMIC-small, we extract several smaller samples from the data generated using the synthea framework.
Tables 1 and 2 provide a summary of patient characteristics in the pretraining and finetuning datasets, respectively.
Table 1.
Descriptive summary of the pretraining dataset.
| Characteristic | Pretraining data |
|---|---|
| Total number of patients | 84 431 |
| Average number of visits per patient | 3 |
| Average number of codes per patient | 23 |
| Vocabulary size | 27 052 |
Table 2.
Descriptive summary of the finetuning datasets.
| Outcome characteristic | Finetuning data |
||||
|---|---|---|---|---|---|
| MIMIC | MIMIC-small | SIPPS | Synthea-small | ||
| SUD | Total number of patients | 77 341 | 2650 | 2182 | 2650 |
| Number of patients with event | 6984 | 750 | 619 | 750 | |
| Average number of visits per patient | 4 | 4 | 22 | 14 | |
| Average number of codes per patient | 28 | 29 | 30 | 12 | |
| Vocabulary size | 20 196 | 6704 | 5556 | 189 | |
| OUD | Total number of patients | 83 354 | 2780 | 2364 | 1743 |
| Number of patients with event | 1585 | 130 | 110 | 83 | |
| Average number of visits per patient | 4 | 4 | 28 | 16 | |
| Average number of codes per patient | 30 | 29 | 39 | 12 | |
| Vocabulary size | 21 667 | 6800 | 6856 | 186 | |
| Diabetes | Total number of patients | 34 295 | 2182 | 2192 | 2182 |
| Number of patients with event | 2201 | 619 | 459 | 619 | |
| Average number of visits per patient | 4 | 4 | 22 | 10 | |
| Average number of codes per patient | 25 | 29 | 31 | 7 | |
| Vocabulary size | 16 670 | 5933 | 5790 | 151 | |
The statistics are derived from the data post the processing stage
Approaches
We first describe how we adapted some previous methods and then outline our 2 proposed approaches.
Previous methods
MedBERT 6 is an adaptation of the original BERT framework on structured EHR and was trained on diagnoses data coded using the ICD codes. While the pre-trained model and the original data were not available, we made use of the framework and pretrained it using the MIMIC data. MedBERT was chosen because of its popularity, ease of implementation, and performance. For more details, please refer to the paper.6
GRAM 3 supplements structured EHR with hierarchical information inherent to medical ontologies. Its implementation allows for the assessment of the effectiveness of a domain knowledge integration-based approach and a general deep learning model, without pretraining, in handling various data types.
Logistic regression & balanced random forest were used as baselines for simpler models. We selected the top 10% of the most frequent diagnosis codes among patients with the target condition as predictors and used multi-hot encoding to represent these predictors.
Sent-e-Med
Sent-e-Med is a direct modification of MedBERT. Any other LM designed explicitly for structured EHR could have been viable as well. As depicted in Figure 3, we represent each visit as a variable-length sequence of diagnosis codes, which are then mapped to textual descriptions and encoded using the pretrained S-BERT25 to create “sentence embeddings.” Unlike MedBERT, these embeddings in Sent-e-Med are kept frozen throughout pretraining and fine-tuning to help transfer valuable knowledge about diagnostic associations, improving model efficiency and performance. We also tried not freezing the sentence embeddings but did not obtain satisfactory results.
Figure 3.
High-level overview of the Sent-e-Med architecture: for each medical code, sentence embeddings and visit embeddings are extracted and subsequently combined before being fed into the transformer encoder as input.
Additionally, “visit embeddings” serve as unique identifiers for each visit, akin to segment embeddings in BERT and are randomly initialized and updated during pretraining. Similar to MedBERT, we do not use [CLS] and [SEP] tokens and omit positional encodings due to format differences between EHR data and standard text. The final input embedding is generated by summing the sentence and visit embeddings and is passed to the transformer encoder.
To enhance its suitability for risk prediction tasks, Sent-e-Med introduces a new pretraining objective: next-visit prediction. Once the model is pretrained, a classification layer (fully-connected layer) is added on top of the model for finetuning on the desired tasks.
More details on this approach and its benefits are discussed in the Supplementary File.
LLaMA2-EHR
We finetune the 7B-chat version of the LLaMA217 model, one of the top open-source LLMs,26 on structured EHR and dub it LLaMA2-EHR (There is a larger, better-performing version (70B) available, however it is impractical to use in most settings due to limited computational resources). To facilitate a more straightforward comparison with the other approaches, we finetune the model exclusively on diagnosis history.
The structure of our data differs from the original datasets used for training LMs. Thus, we explore 2 distinct approaches for constructing the input data for LLaMA2-EHR: (1) cumulatively aggregating all visit information to provide a summary of a patient’s history (Prompt 1—Figure 4A), (2) utilizing temporal information to represent patient visits (Prompt 2—Figure 4B).
Figure 4.
Illustration of 2 distinct prompts employed in the fine-tuning of the LLaMA2-EHR model. Prompt 1 aggregates the frequency of diagnosis occurrences across multiple visits, while Prompt 2 evaluates diagnoses on a per-visit basis and incorporates information about the intervals between visits. Red highlights in the text are employed to indicate patient-specific variations in the information.Note: Inputs in the prompts and the responses are just hypothetical examples.
Unlike Sent-e-Med, which requires pretraining on the EHR corpus while only keeping the sentence embeddings frozen, LLaMA2-EHR focuses solely on fine-tuning for a limited number of epochs.
Benefits of text-based representation
Both Sent-e-Med and LLaMA2-EHR operate without dependency on specific coding systems. In the healthcare domain, different organizations may use distinct vocabularies to encode medical entities, such as SNOMED CT or ICD-10 for diagnosis. Even within the same organization, the coding system undergoes frequent changes, for example, transitioning from ICD-9 to ICD-10. By operating on textual descriptions, these models mitigate concerns about the specific vocabularies present in the data, enabling its application across diverse datasets.
Results
Evaluation
Considering our risk prediction tasks, we prioritized assessing probability scores over binary labels (0 and 1). To better understand the models’ effectiveness in distinguishing patients based on varying risk probabilities, we chose ROC AUC and PR AUC as our preferred evaluation metrics. Unlike other models, assessing LLaMA2 for classification poses a unique challenge due to its verbose text output. To address this, we use the input up to the XML tag <Diagnosis> and obtain the probability distribution of the possible next tokens. Ideally, this stage should yield high probabilities for tokens “Yes” or “No.” However, another challenge arises from the multiple representations of “Yes” and “No.” For example, “Yes” was represented as either “Y” and “es” separately, “YES,” “Yes,” or “yes.” To tackle this, we first identify the potential representations of “Yes” and “No” by analyzing the outputs over multiple runs. Then, we take the tokens with the top-k probability scores (we used k = 20), sum those associated with potential representations of “Yes” and “No” separately, and then apply a softmax over the 2 values to obtain the final representations.
Comparative analysis
As outlined in Materials and methods section, we experimented with 4 different types of data for 3 separate tasks during finetuning: OUD, SUD, and Diabetes risk prediction. We used the same train, test, and validation sets for all models for fair comparison. The MIMIC dataset, which was used for pretraining, has a pre-separated training, test and validation sets. Therefore, we performed a single set cross validation on MIMIC. In contrast, we performed a 5-fold cross-validation on SIPPS and Synthea. Table 3 summarizes the results we obtained across our test sets. More detailed results with standard errors are presented in the Supplementary File.
Table 3.
Test ROC AUC and PR AUC scores across our datasets in the context of our prediction tasks.
| Outcome | Model | MIMIC |
MIMIC-small |
SIPPS |
Synthea-small |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ROC AUC | PR AUC | BR | ROC AUC | PR AUC | BR | ROC AUC | PR AUC | BR | ROC AUC | PR AUC | BR | ||
| SUD | Random Forest | 0.81 | 0.34 | 0.1 | 0.85 | 0.67 | 0.4 | 0.74 | 0.57 | 0.4 | 0.77 | 0.59 | 0.4 |
| Logistic | 0.81 | 0.32 | 0.1 | 0.75 | 0.62 | 0.4 | 0.74 | 0.57 | 0.4 | 0.79 | 0.63 | 0.4 | |
| GRAM | 0.85 | 0.35 | 0.1 | 0.87 | 0.65 | 0.4 | 0.74 | 0.54 | 0.4 | 0.86 | 0.73 | 0.4 | |
| MedBERT | 0.89 | 0.70 | 0.1 | 0.87 | 0.77 | 0.4 | 0.75 | 0.57 | 0.4 | 0.87 | 0.75 | 0.4 | |
| Sent-e-Med | 0.91*** | 0.73 *** | 0.1 | 0.89* | 0.79 | 0.4 | 0.76 | 0.58 | 0.4 | 0.87 | 0.76 | 0.4 | |
| LLaMA2-EHR | 0.92 *** | 0.73 *** | 0.1 | 0.93 *** | 0.89 *** | 0.4 | 0.74 | 0.56 | 0.4 | 0.88 ** | 0.78 ** | 0.4 | |
| OUD | Random Forest | 0.87 | 0.18 | 0.02 | 0.71 | 0.34 | 0.05 | 0.75 | 0.17 | 0.05 | 0.88 | 0.33 | 0.05 |
| Logistic | 0.86 | 0.19 | 0.02 | 0.83 | 0.39 | 0.05 | 0.75 | 0.17 | 0.05 | 0.88 | 0.33 | 0.05 | |
| GRAM | 0.90 | 0.22 | 0.02 | 0.87 | 0.30 | 0.05 | 0.74 | 0.17 | 0.05 | 0.92 | 0.57 | 0.05 | |
| MedBERT | 0.91 | 0.53 | 0.02 | 0.89 | 0.57 | 0.05 | 0.74 | 0.19 | 0.05 | 0.93 | 0.6 | 0.05 | |
| Sent-e-Med | 0.95 *** | 0.58* | 0.02 | 0.93 | 0.57 | 0.05 | 0.75 | 0.19 | 0.05 | 0.95 | 0.7 | 0.05 | |
| LLaMA2-EHR | 0.95 *** | 0.59 *** | 0.02 | 0.95 ** | 0.72 *** | 0.05 | 0.73 | 0.2 | 0.05 | 0.98 *** | 0.84 *** | 0.05 | |
| Diabetes | Random Forest | 0.75 | 0.22 | 0.07 | 0.75 | 0.58 | 0.4 | 0.82 | 0.59 | 0.27 | 0.88 | 0.74 | 0.4 |
| Logistic | 0.72 | 0.19 | 0.07 | 0.74 | 0.5 | 0.4 | 0.81 | 0.59 | 0.27 | 0.88 | 0.76 | 0.4 | |
| GRAM | 0.77 | 0.16 | 0.07 | 0.71 | 0.4 | 0.4 | 0.71 | 0.52 | 0.27 | 0.93 | 0.86 | 0.4 | |
| MedBERT | 0.84 | 0.61 | 0.07 | 0.8 | 0.72 | 0.4 | 0.79 | 0.57 | 0.27 | 0.94 | 0.86 | 0.4 | |
| Sent-e-Med | 0.89 *** | 0.67 * | 0.07 | 0.83 ** | 0.74*** | 0.4 | 0.81*** | 0.62*** | 0.27 | 0.94 | 0.88 ** | 0.4 | |
| LLaMA2-EHR | 0.88*** | 0.67 * | 0.07 | 0.83 * | 0.76 ** | 0.4 | 0.83 *** | 0.64 *** | 0.27 | 0.94 | 0.88 * | 0.4 | |
BR represents the base rate, that is, the ratio of positive to negative samples. The bolded values represent the best performance across models.
For Sent-e-Med and LLaMA2-EHR, values marked with asterisks indicate a significant deviation from the best baseline performance. Significance is assessed using a bootstrap test, resampling 1000 times.
P < .1,
P < .05,
P < .01.
We omit the results obtained from using Prompt 2 on LLaMA2-EHR due to its suboptimal performance on SIPPS. The LLaMA217 model’s sequence length is limited to 4096 tokens which cannot accommodate longer input sequence resulting from high number of visits in SIPPS (see Table 2). For other datasets, the results closely resembled those obtained from Prompt 1.
The following are the main takeaways from the results shown in Table 3:
Among the baselines, MedBERT appears to exhibit the best performance in the majority of the tested tasks and datasets.
The pretraining-based methods (ie, MedBERT, Sent-e-Med, and LLaMA2-EHR) are significantly better than other methods in terms of both metrics in all but one dataset (SIPPS). In the SIPPS dataset, all of the experimented models had almost similar performances, except for the diabetes risk prediction task.
Across all tasks and datasets, LLaMA2-EHR consistently demonstrated superior (or at least equal) performance compared to the other models. This is particularly noteworthy for the PR AUC values. For example, in the OUD prediction task, LLaMA2-EHR achieved a PR AUC of 72%, while MedBERT scored 57% on the MIMIC-small dataset. To note, we also finetuned a Clinical-Longformer27 model. While this model performed comparably to Sent-e-Med, it did not perform as well as LLaMA2-EHR.
Overall, the models that utilized the textual inputs (Sent-e-Med and LLaMA2-EHR) had better performances as compared to others.
Between the 2 variants of Sent-e-Med, both demonstrated nearly identical performance, although the variant that only used the MLM training objective performed slightly better overall. We hypothesize that introducing the next visit prediction objective added more complexity to the model making it harder to learn effective representations. Consequently, we just present results from the model utilizing only the MLM objective.
In most cases, simpler models such as logistic regression and random forest did not exhibit strong performance. Discussion section delves into the potential explanations for these observations and explores more noteworthy findings.
LLaMA2-EHR sensitivity analysis
Changing the instructions
Our results have indicated that LLaMA2-EHR generally performs well when tested with the same instructions used during the fine-tuning stage. However, we were also interested in exploring what happens when we make slight alterations to the instructions. For instance, what if we modify the instruction to predict the risk (High/Low) of having a diagnosis, as opposed to asking a Yes/No question? We tested 2 different models. The first model, finetuned on the MIMIC dataset, consistently failed to adhere to the provided instructions, outputting “Yes” or “No” rather than “High” or “Low” in all instances. The second model, finetuned on the MIMIC-small dataset, exhibited the desired behavior, predicting “High” and “Low” as instructed in most cases. However, in approximately 35% of the instances, the outputs were inverted, meaning it predicted “High” when the output with the original prompt was “No” and “Low” when the output was “Yes.”
We also conducted this experiment using the original LLaMA2-7B-chat model and found that the model could generally adhere to and understand the modified instruction, yielding conforming probabilities of High/Yes and Low/No.
This discrepancy raises concerns about the adaptability of the model to changing instructions. It suggests that fine-tuning on our datasets might have induced catastrophic forgetting28 particularly when dealing with a large volume of data.
Changing the inputs
We investigated the sensitivity of the models with respect to changes in the input data. We initially created a simple input containing historical diagnosis information for a hypothetical patient (see Figure 5A) and directed the model to predict whether the patient would develop Diabetes in the future using Prompt 1. Subsequently, we made slight adjustments to the input by including additional past diagnoses recognized as risk factors for Diabetes (as illustrated in Input 2 in Figure 5B). Ideally, we would expect the probability of Yes to increase while doing so. With the MIMIC finetuned model, the probability increased from 3% to 5%. With the MIMIC-small finetuned model, the probability increased from 2% to 7%, a slightly more pronounced change. It is challenging to evaluate these numbers without predetermined benchmarks for the expected increase. Nonetheless, both models did exhibit some level of responsiveness to changes in the input data.
Figure 5.
Examining the variations in LLaMA2-EHR responses when predicting the probability of Diabetes diagnosis based on 2 distinct inputs: one representing a simple hypothetical patient’s medical history (Input 1) and another involving additional diagnosis information that is known to be the risk factors of Diabetes (Input 2). The objective is to analyze how the likelihood of a “Yes” or “No” prediction changes within these specific scenarios.
We conducted a similar experiment with MedBERT and Sent-e-Med. Interestingly, the output probabilities in MedBERT did not change as anticipated; instead, there was an opposite effect. In contrast, with Sent-e-Med, the probability of Yes did increase for Input 2. This initial experiment suggests that the proposed models may exhibit greater responsiveness to changes in the input.
Discussion
In this study, we conducted a comparative examination of various methods for structured EHR-based risk prediction. Additionally, we introduced 2 novel approaches (Sent-e-Med and LLaMA2-EHR) utilizing LMs in this context. To our knowledge, this is the first comprehensive study applying LLMs to structured EHRs for risk prediction.
Overall, our observations indicate that LM-based methods, including MedBERT, Sent-e-Med, and LLaMA2-EHR, consistently outperformed the other 3 baseline approaches across the majority of the datasets. This trend aligns with prior research findings,4,7,29 highlighting the significant role of pretraining in enhancing performance in downstream tasks. In contrast, simpler models like Random Forest and Logistic Regression may not have been able to capture the intricate longitudinal aspects within EHRs. GRAM, on the other hand, captures such information while providing other benefits such as domain knowledge integration. This model showed comparable performances on Synthea-small but did not perform as well on the other datasets. Deep learning methods typically demand substantial data to acquire effective representations. This requirement was likely unmet in these datasets. For instance, even in the MIMIC dataset, the number of positive samples is quite low while the vocabulary size remains substantial. In contrast, Synthea-small had a relatively smaller vocabulary size, as evidenced in Table 2.
In the SIPPS dataset, all the experimented models exhibited nearly indistinguishable performance, except in the Diabetes prediction task. This dataset mostly consists of outpatient visits and notably features a higher average number of visits (Table 2), which could present challenges in capturing its complexity due to the dataset’s limited size.
The MIMIC-small and Synthea-small datasets were primarily introduced to discern whether the performance on SIPPS was influenced by differences in domain or data quality, as opposed to the small sample size. As depicted in Table 3, the pretraining-based approaches, especially Sent-e-Med and LLaMA2-EHR, outperform the previous approaches on these datasets. This suggests that these approaches perform well even with a limited amount of target data.
Among the LM-based approaches, those that leverage textual descriptions instead of relying solely on medical ontology (Sent-e-Med and LLaMA2-EHR) appear to have harnessed the knowledge encoded within LMs more effectively. For instance, in the Synthea-small dataset, specifically in the OUD prediction task, Sent-e-Med and LLaMA2-EHR exhibit significantly superior performance, evident in higher PR AUC values compared to MedBERT. One possible explanation for this difference might be attributed to the presence of around 8% previously unseen diagnosis concepts during fine-tuning, absent in the initial pretraining phase. Unlike MedBERT’s random embedding initialization for new codes, these methods leverage knowledge from their pretrained representations. The more pronounced performance difference in Synthea-small may be attributed to the relatively limited number of positive samples. To test this, we conducted an experiment with Synthea-small, categorizing it into 2 groups: one with infrequent concepts and one without. We found a more significant performance gap between the proposed methods (LLaMA-2 EHR and Sent-e-Med) and MedBERT, especially in the group with infrequent concepts (results in Supplementary File).
Notably, LLaMA2-EHR exhibited the most impressive overall performance, which aligns with our expectations, given that LLaMA2 stands as one of the best LMs currently available and already encompasses a wealth of valuable information that can be advantageous for downstream tasks. The performance difference was much more prominent in the PR AUC values. While ROC AUC values offer insight into the overall separability of data points, PR AUC values place special emphasis on positive examples. When dealing with data imbalances, the examination of both these metrics becomes crucial, as differences between them tend to become more pronounced as shown in Table 3. The substantially higher PR AUC values achieved by the LLaMA2-EHR model compared to baselines suggest that this approach excels even in the presence of data imbalance, as it prioritizes both positive and negative examples effectively. Furthermore, LLaMA2-EHR offers a notably more straightforward approach, especially in comparison to MedBERT and Sent-e-Med, by solely necessitating fine-tuning the EHR corpus. We also tested the LLaMA2 model directly without finetuning, however, the results were not satisfactory. For instance, an ROC AUC score of 50% and a PR AUC score of 29% was obtained in the MIMIC diabetes risk prediction task.
Conclusions
To sum up, we conducted a thorough examination of structured EHR-based risk prediction methods, introducing 2 novel approaches utilizing LMs. The consistent outperformance of LM-based methods, particularly LLaMA2-EHR, highlights the substantial impact of utilizing pretrained knowledge in enhancing performance across datasets. Nevertheless, the study emphasizes the importance of exercising caution when employing LMs due to potential issues with prompt sensitivity and knowledge retention during fine-tuning, among other concerns.30 Overall, this study provides valuable insights into the strengths and limitations of LM-based approaches in EHR-based risk prediction.
Our study primarily focused on a retrospective analysis of prior diagnoses, driven by the specific goal of conducting a comparative analysis within a targeted data scenario rather than maximizing the use of available information. We acknowledge that conditions like SUD, OUD, and Diabetes might necessitate a more comprehensive exploration of patient history, including factors like medications, lab results, or social determinants of health for SUD and OUD. Future research could explore the incorporation of such additional inputs. Notably, an extended context length may be required for LMs to accommodate all the pertinent information, but there are viable solutions available for this purpose.31,32
Supplementary Material
Contributor Information
Angeela Acharya, George Mason University, Fairfax, VA, United States.
Sulabh Shrestha, George Mason University, Fairfax, VA, United States.
Anyi Chen, Staten Island Performing Provider System, Staten Island, NY, United States.
Joseph Conte, Staten Island Performing Provider System, Staten Island, NY, United States.
Sanja Avramovic, George Mason University, Fairfax, VA, United States.
Siddhartha Sikdar, George Mason University, Fairfax, VA, United States.
Antonios Anastasopoulos, George Mason University, Fairfax, VA, United States.
Sanmay Das, George Mason University, Fairfax, VA, United States.
Author contributions
Angeela Acharya (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing—original draft, Writing—reviewing and editing), Sulabh Shrestha (Conceptualization, Methodology, Software, Visualization, Writing—reviewing and editing), Anyi Chen (Conceptualization, Data curation, Resources, Writing—reviewing and editing), Joseph Conte (Conceptualization, Data curation, Resources, Reviewing), Sanja Avramovic (Conceptualization, Resources, Validation, Writing—reviewing and editing), Siddhartha Sikdar (Conceptualization, Resources, Supervision, Funding Acquisition, Reviewing), Antonios Anastasopoulos (Conceptualization, Methodology, Supervision, Validation, Writing—reviewing and editing), and Sanmay Das (Conceptualization, Methodology, Supervision, Validation, Project Administration, Writing—original draft, Writing—reviewing and editing)
Supplementary material
Supplementary material is available at Journal of the American Medical Informatics Association online.
Funding
This research was supported in part by the NSF grants 1945764 (PI: Sikdar) and 2327143 (PI: Anastasopoulos). The GPU resources were provided by the Office of Research Computing (https://orc.gmu.edu) at George Mason University and funded in part by NSF grants 1625039 and 2018631.
Conflicts of interest
The authors have no competing interests to declare.
Data availability
The MIMIC-IV dataset is publicly available to use upon completing the necessary trainings (https://physionet.org/content/mimiciv/2.2/). The Synthea-small dataset can be made available upon request. Due to privacy and ethical considerations, the SIPPS dataset is not openly shared or published. However, qualified researchers who wish to access the data for legitimate research purposes may contact the corresponding author for data sharing inquiries.
References
- 1. Pendergrass SA, Crawford DC. Using electronic health records to generate phenotypes for research. Curr Protoc Hum Genet. 2018;100(1):e80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Goldstein BA, Navar AM, Pencina MJ, et al. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc. 2017;24(1):198––208.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Choi E, Bahadori MT, Song L, et al. Gram: Graph-based attention model for healthcare representation learning. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17. Association for Computing Machinery. 2017:787–795. [DOI] [PMC free article] [PubMed]
- 4. Shang J, Ma T, Xiao C, et al. Pre-training of graph augmented transformers for medication recommendation. CoRR, abs/1906.00346. 2019.
- 5.Hirsch JA, Nicola G, McGinty G, et al. ICD-10: History and Context. AJNR Am J Neuroradiol. 2016;37(4):596–599. 10.3174/ajnr.A4696 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Rasmy L, Xiang Y, Xie Z, et al. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Digit Med. 2021;4(1):86. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Prakash P, Chilukuri S, Ranade N, et al. RareBERT: transformer architecture for rare disease patient identification using administrative claims. AAAI. 2021;35(1):453––460.. [Google Scholar]
- 8. Li Y, Rao S, Solares JRA, et al. BEHRT: transformer for electronic health records. Sci Rep. 2020;10(1):7155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Pang C, Jiang X, Kalluri KS, et al. CEHR-BERT: incorporating temporal information from structured EHR data to improve prediction tasks. In: Proceedings of Machine Learning for Health, Volume 158 of Proceedings of Machine Learning Research. PMLR; 2021:239–260. [Google Scholar]
- 10. Devlin J, Chang M-W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics; 2019.
- 11. Luo F, Li M, Florence C. State-level economic costs of opioid use disorder and fatal opioid overdose–United States, 2017. MMWR Morb Mortal Wkly Rep. 2021;70(15):541––546.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Acharya A, Sikdar S, Das S. Gensyn: A multi-stage framework for generating synthetic microdata using macro data sources. In: 2022 IEEE International Conference on Big Data (Big Data). 2022:685–692.
- 13. Acharya A, Izquierdo AM, Gonçalves SF, et al. Exploring county-level spatio-temporal patterns in opioid overdose related emergency department visits. PLoS One. 2022;17(12):e0269509. 12 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. McLellan AT. Substance misuse and substance use disorders: Why do they matter in healthcare? Trans Am Clin Climatol Assoc. 2017;128:112––130.. [PMC free article] [PubMed] [Google Scholar]
- 15. Huang K, Altosaar J, Ranganath R. Clinicalbert: modeling clinical notes and predicting hospital readmission. CoRR. abs/1904.05342, 2019.
- 16. Yang X, Chen A, PourNejatian N, et al. Gatortron: a large clinical language model to unlock patient information from unstructured electronic health records. npj Digit Med. 2022;5:194. [DOI] [PMC free article] [PubMed]
- 17. Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models. arXiv. 2023.
- 18. Chowdhery A, Narang S, Devlin J, et al. PaLM: scaling language modeling with pathways. J Mach Learn Res. 2023;24(240):1–113. [Google Scholar]
- 19. Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, eds., Advances in Neural Information Processing Systems, Vol 33. Curran Associates, Inc., 2020:1877–1901. [Google Scholar]
- 20. Johnson A, Bulgarelli L, Pollard T, et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci Data. 2023;10:1. 10.1038/s41597-022-01899-x [DOI] [PMC free article] [PubMed]
- 21. Walonoski J, Kramer M, Nichols J, et al. Synthea: an approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. J Am Med Inf Assoc. 2017;25(3):230––238.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Gaudet-Blavignac C, Foufi V, Bjelogrlic M, et al. Use of the systematized nomenclature of medicine clinical terms (SNOMED CT) for processing free text in health care: systematic scoping review. J Med Internet Res. 2021;23(1):e24594. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Shickel B, Tighe P, Bihorac A, et al. Deep EHR: a survey of recent advances on deep learning techniques for electronic health record (EHR) analysis. CoRR, abs/1706.03446, 2017. [DOI] [PMC free article] [PubMed]
- 24. Kashyap A, Callison-Burch C, Boland MR. A deep learning method to detect opioid prescription and opioid use disorder from electronic health records. Int J Med Inf. 2023;171:104979. 10.1016/j.ijmedinf.2022.104979 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics; 2019:3982–3992. [Google Scholar]
- 26. Naveed H, Khan AU, Qiu S, et al. A comprehensive overview of large language models. arXiv. 2023.
- 27. Li Y, Wehbe RM, Ahmad FS, Wang H, Luo Y. Clinical-longformer and clinical-bigbird: transformers for long clinical sequences. CoRR, abs/2201.11838, 2022.
- 28. Kaushik P, Gain A, Kortylewski A, et al. Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping. CoRR, abs/2102.11343, 2021.
- 29. Shrestha S, Li Y, Kosecka J. Self-supervised pre-training for semantic segmentation in an indoor scene. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops. 2024:625–635.
- 30. McKenna N, Li T, Cheng L, et al. Sources of hallucination by large language models on inference tasks. In: Findings of the Association for Computational Linguistics. EMNLP; 2023:2758–2774. [Google Scholar]
- 31. Peng B, Quesnelle J, Fan H, et al. YaRN: efficient context window extension of large language models. arXiv. 2023.
- 32. Chen S, Wong S, Chen L, et al. Extending context window of large language models via positional interpolation. arXiv. 2023.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The MIMIC-IV dataset is publicly available to use upon completing the necessary trainings (https://physionet.org/content/mimiciv/2.2/). The Synthea-small dataset can be made available upon request. Due to privacy and ethical considerations, the SIPPS dataset is not openly shared or published. However, qualified researchers who wish to access the data for legitimate research purposes may contact the corresponding author for data sharing inquiries.