Large-Scale Automated Phenotyping of Cardiac Arrest and Withdrawal of Life-Sustaining Therapy Using Electronic Health Record Data

Abstract

Aims of the Study:

Anoxic brain injury following cardiac arrest is a leading cause of death in the United States. Withdrawal of life-sustaining therapy (WLST) is a common end-of-life decision in these patients, but its contributing factors and outcomes remain poorly understood. We developed machine learning models to enable large-scale, automated phenotyping to identify patients who died following WLST.

Methods:

We used structured and unstructured EHR (Electronic Health Record) data from two major hospitals to train models that identify (1) patients with cardiac arrest and coma, and (2) patients who died after WLST. Performance was evaluated using the area under the receiver operating characteristic (AUROC) and precision-recall (AUPRC) curves, as well as other precision metrics.

Results:

On holdout (internal) testing the models achieved AUROC/AUPRC values of 0.984/0.968 (cardiac arrest) and 0.992/0.991 (WLST). Cross-hospital evaluation showed strong performance for the cardiac arrest phenotype but variable generalizability for the WLST phenotype, with sensitivity depending on the training site.. Population-level error rates were low (<0.5%) for the cardiac arrest phenotype; estimates for WLST varied by hospital.

Conclusion:

These models establish a reproducible framework for automated cohort identification. Nearly half of comatose post-arrest patients died following WLST, with 42% of these deaths occurring within 72 hours, highlighting the impact of early prognostication decisions. The models enable rapid cohort identification for research on neuroprognostication, including how WLST decisions may perpetuate self-fulfilling prophecies. Broader validation across health systems and larger cohorts will improve generalizability and inform evidence-based end-of-life decision-making.

Introduction

About 80% of cardiac arrest patients remain comatose, with many failing to regain consciousness for days.^1,2,3 Accurate identification of postarrest coma and withdrawal of life-sustaining therapy (WLST) from electronic health records (EHRs) is essential for understanding practice patterns and outcomes after cardiac arrest. Withdrawal of life-sustaining therapy (WLST) is frequently chosen in cases where recovery may still be possible, confounding outcome prediction and contributing to “self-fulfilling prophecies”, where poor outcomes occur because they were anticipated.⁴

Existing registries of data such as International Classification of Diseases (ICD) codes incompletely capture neurologic outcomes, and manual chart review is infeasible at scale.^5,6,7,8 Automated phenotyping using unstructured note data can fill this gap by simulating manual review in exponentially shorter time if methods are transparent and generalizable.^{9, 10}

We present a reproducible phenotyping pipeline that achieves high accuracy on internal holdout data and demonstrates feasibility for cross-hospital use, while highlighting limitations in cross-site generalizability (particularly for WLST) that warrant further multicenter validation. This model lays the groundwork for studying how WLST decisions are made and how they may be influenced by prior cases, potentially contributing to self-fulfilling prognostic cycles.¹¹

Methods

Cohort construction

We used records from the Mass General Brigham (MGB; including Massachusetts General and Brigham and Women’s Hospitals) deidentified institutional database (2013–2023), and Beth Israel Deaconess Medical Center’s (BIDMC) publicly available MIMIC-III database (2001–2012).^12,13,14

Data sources included physician discharge summaries, International Classification of Diseases (ICD) codes, and Current Procedural Terminology (CPT) codes. Only discharge summaries exceeding 500 words were included to ensure sufficient clinical detail. To balance positive and negative cases for training, we created an enriched cohort of 3,000 unique patients using ICD and CPT codes associated with cardiac arrest and neurological injury, selected per clinical guidelines and expert consensus.¹⁴ Detailed selection criteria and code lists are provided in Supplemental Section S1 and Table S1

Determining ground truth

A standardized case-classification protocol was developed in collaboration with four board-certified neurologists (M.B.W., E.D.A., S.F.Z., S.S.M.). Cases were labeled positive if documentation indicated both cardiac arrest and coma. To avoid bias, annotators were blinded to algorithmic predictions and hospital of origin during labeling. The full manual annotation procedure is detailed in Supplemental Table S2.

For Model 2, the positive cases from Model 1 served as input for WLST classification. WLST determinations followed a protocol reviewed by two neurologists (M.B.W., E.A.). A case was classified as WLST only if the patient died during hospitalization and death was explicitly attributed to withdrawal of medical care. Deaths from other causes (e.g., brain death) or WLST without death were excluded. Among 938 cases meeting criteria for cardiac arrest and coma (740 [78.9%] from MGB; 198 [21.1%] from BIDMC), 444 (47.3%) were identified as WLST. Time to WLST was approximated as the interval between admission and discharge.

Matrix Creation

Features for Model 1 were derived from ICD codes and keywords extracted from clinical notes. During manual annotation, influential terms were recorded as candidate features. Text preprocessing included stemming and negation handling using standard NLP tools. Keywords occurring fewer than 30 times across the entire dataset were excluded.¹⁵ Features were used to construct a binary feature matrix, with each feature coded as “1” if present and “0” if absent. Negated terms were treated as separate features; if both a term and its negation appeared, both were assigned a “1.” For additional information see Supplemental Section S2 and Supplemental Table S3.

Model Development and Validation

Binary feature matrices served as inputs for predictive modeling. Model 1 used 3,000 patients split 1,500/1,500; Model 2 used 938 patients split 469/469. Splits were fixed prior to tuning and testing data were never used for model selection. For external validation, a single model trained on one hospital’s training set was applied to the other hospital’s holdout set, ensuring no patient overlaps.

We compared random forest and logistic regression models using ICD-only and ICD + keyword feature sets to assess the value of unstructured text. Binary cut-points were chosen by maximizing Youden’s J on the training set and applied unchanged to testing and external data. The final model was trained on the entire training set after hyperparameter tuning by cross-validation on training folds‥¹⁶ Model optimization procedures are detailed in Supplemental Section S3.

Cross-Hospital Generalizability Analysis

To test generalizability of model 1, a model was trained on BIDMC notes from the Master Training Set (n=511) and tested on MGB notes from the Master Holdout Set (n=1011). This was repeated with MGB (n=989) training and BIDMC (n=489) testing. This was repeated for model 2 with MGB training (n=366) and BIDMC testing (n=104), and with BIDMC training (n=94) and MGB testing (n=374). We acknowledge that the available sample sizes for this nested Model 2 analysis (e.g., n=94 for the BIDMC training set) are below standard recommendations for machine learning derivation. Consequently, this cross-hospital assessment was undertaken as an exploratory analysis to investigate the impact of training size and site-specificity on model performance.

Error Analysis

After final model evaluation all misclassified cases were manually reviewed to determine the sources of error. A cohort reconstruction was conducted using 40,000 patients (20,000 patients from each hospital), to estimate the error rate in a randomly selected population.

The probability of classification error (Pe) was calculated using the following formula:

$$Pe=(P{e}_{++}·p_{++})+(P{e}_{+-}·p_{+-})+(P{e}_{-+}·p_{-+})+(P{e}_{--}·p_{-})$$

where Pe represents the overall error probability, and p the proportion of cases in each classification group.

Results

Cohort Construction

The final study cohort included 3000 unique patients, with 2000 from MGB and 1000 from BIDMC. It contained a relatively higher number of males (57.4%). The most common race in the cohort was White, followed by Black/African American, Asian, and American Indian or Alaska Native, with 6.8% of patients identifying as Hispanic. Race was unavailable for 13.3% of the cohort. See Table 1 for cohort demographic specifics. Supplemental tables S5 and S6 contain further details about cohort creation.

Table 1:

Patient demographics for MGB and BIDMC. Included are the number of participants, the average age including one standard deviation from the mean race, and ethnicity.

Hospital	ICD+, CPT+	ICD+, CPT−	ICD−, CPT+	ICD−, CPT−
Participants
MGB	500	500	500	500
BIDMC	250	250	250	250
Average Age (St. Dev.)
MGB	57.9 (18.9)	59.3 (18.1)	56.5 (21.9)	55.4 (23.6)
BIDMC	62.2 (15.3)	68.7 (14.8)	62.1 (18.3)	60.69 (23.8)
Female (%)
MGB	166 (33.2%)	184 (36.8%)	228 (45.6%)	256 (51.2%)
BIDMC	89 (35.6%)	123 (49.2%)	110 (44.0%)	121 (48.4%)
White (%)
MGB	329 (65.8%)	346 (69.2%)	369 (73.8%)	378 (75.6%)
BIDMC	175 (70.0%)	179 (71.6%)	195 (78.0%)	183 (73.2%)
African American (%)
MGB	(16.2%)	(13.4%)	(10.8%)	(7.6%)
BIDMC	(14.0%)	(14.4%)	(10.0%)	(5.2%)
Asian (%)
MGB	(2.0%)	(3.4%)	(2.2%)	(4.4%)
BIDMC	(3.2%)	(1.6%)	(3.2%)	(3.2%)
American Indian or Alaska Native (%)
MGB	(0.6%)	(0.4%)	(0.2%)	(0.2%)
BIDMC	(0.4%)	(0.0%)	(0.0%)	(0.4%)
Unavailable/Other (%)
MGB	(15.4%)	(13.4%)	(12.8%)	(12.2%)
BIDMC	(13.2%)	(12.4%)	(8.8%)	(18.0%)
Hispanic (%)
MGB	(9.8%)	(10.0%)	(6.6%)	(7.4%)
BIDMC	(2.8%)	(4.8%)	(4.4%)	(2.4%)

Model Performance

Random forest models consistently outperformed logistic regression and were selected for final evaluation. Detailed performance metrics are provided in Supplemental Table S4, with training ROC and PR curves by fold in Supplemental Figures S1-S3.This final model is a single model trained on all training data after hyperparameter selection. 95% CIs for binary metrics used Clopper–Pearson exact method; AUROC/AUPRC CIs were derived from 1,000 bootstrap resamples. See table 2 for performance metrics and Figure 1 for AUROC and AUPRC curves. The model demonstrated high specificity (e.g., 96.1%) and positive predictive value (PPV) (e.g., 91.4%), indicating a low false positive rate and a high degree of reliability for positive classifications on the holdout set.

Table 2:

AUROC, AUPRC, and F1 scores for evaluating performance, and sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and confusion matrix for evaluating practical utility of final models.

	Model 1	Model 2
AUROC (95% CI)	0.984 (0.979–0.989)	0.992 (0.9857–0.996)
AUPRC (95% CI)	0.968 (0.958–0.977)	0.991 (0.984–0.996)
F1 (95% CI)	0.900 (0.880–0.919)	0.955 (0.933–0.973)
Sensitivity: TP/(TP+FN) (95% CI)	0.879 (0.846–0.907)	0.968 (0.935–0.987)
Specificity: TN/(TN+FP)(95% CI)	0.966 (0.953–0.976)	0.950 (0.916–0.973)
PPV: TP/(TP+FP) (95% CI)	0.923 (0.895–0.946)	0.942 (0.903–0.969)
NPV: TN/(TN+FN) (95% CI)	0.945 (0.929–0.958)	0.972 (0.944–0.989)
Confusion Matrix	TP: 420, FP: 35 FN: 58, TN: 987	TP: 212, FP: 13 FN: 7, TN: 246

Figure 1:

AUROC and AUPRC curves from the final models. The solid line is the empirical curve, and the shaded area is the 95% CI.

Feature Importances

Feature importances were calculated to identify which keywords contribute most to prediction of the target phenotype. Afterwards, model training features were reviewed by experts in neurology (M.B.W, E.D.A., S.F.Z, S.S.M.) to confirm that they were reasonable indicators of the phenotypes.

External Validation

External validation was performed by training a single model on the training cases from one hospital and evaluating it on the holdout cases from the other hospital. We did not recalibrate probability thresholds between sites.

For Model 1 performance remained robust across training/testing hospital pairs. For Model 2, performance was variable: when trained on BIDMC and tested on MGB sensitivity was markedly reduced (sensitivity = 0.171; F1 = 0.290), while the reverse train/test pairing yielded substantially better sensitivity (sensitivity = 0.605). These differences appear driven by (1) limited WLST-positive cases in the BIDMC training set, and (2) differences in documentation language between hospitals. We therefore interpret external validation results as demonstrating feasibility of cross-hospital phenotyping for cardiac arrest but limited generalizability for WLST without larger multi-site training data or site-specific recalibration. Full performance metrics are in Table 3.

Table 3:

AUROC, AUPRC, and F1 scores for evaluating performance, and sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and confusion matrix for evaluating practical utility of for external validation step.

Model 1
Training Set/Testing Set:	BIDMC/MGB	MGB/BIDMC
AUROC (95% CI)	0.968 (0.957–0.976)	0.941 (0.917–0.961)
AUPRC (95% CI)	0.951 (0.935–0.964)	0.846 (0.784–0.897)
F1 Score (95% CI)	0.834 (0.804–0.864)	0.703 (0.630–0.766)
Sensitivity: TP/(TP+FN) (95% CI)	0.757 (0.710–0.799)	0.808 (0.719–0.878)
Specificity: TN/(TN+FP) (95% CI)	0.966 (0.948–0.978)	0.868 (0.830–0.900)
PPV: TP/(TP+FP) (95% CI)	0.928 (0.844–0.895)	0.622 (0.535–0.704)
NPV: TN/(TN+FN) (95% CI)	0.871 (0.844–0.895)	0.944 (0.914–0.965)
Confusion Matrix	TP: 283, FP: 22 FN: 91, TN: 615	TP: 84, FP: 51 FN: 20, TN: 334
Model 2
Training Set/Testing Set:	BIDMC/MGB	MGB/BIDMC
AUROC (95% CI)	0.903 (0.871–0.935)	0.888 (0.818–0.948)
AUPRC (95% CI)	0.900 (0.857–0.939)	0.838 (0.706–0.927)
F1 Score (95% CI)	0.290 (0.210–0.234)	0.708 (0.561–0.820)
Sensitivity: TP/(TP+FN) (95% CI)	0.171 (0.119–0.234)	0.605 (0.434–0.760)
Specificity: TN/(TN+FP)(95% CI)	0.990 (0.963–0.999)	0.939 (0.852–0.983)
PPV: TP/(TP+FP) (95% CI)	0.939 (0.798–0.993)	0.852 (0.663–0.958)
NPV: TN/(TN+FN) (95% CI)	0.560 (0.506–0.614)	0.805 (0.699–0.887)
Confusion Matrix	TP: 31, FP: 2 FN: 150, TN: 191	TP: 23, FP: 4 FN: 15, TN: 62

Error Analysis

An error analysis was performed for each model, and incorrect predictions were categorized based on the most common causes. Supplemental Section S4 for the primary sources of error, listed in a decreasing order of prevalence.

Population Error Rates

We estimated the error rate of model 1 in a reconstructed cohort, in which the prevalence of cardiac arrest matches that in the general hospital population. The estimated error rate of model 1 for MGB patients was 0.51%, and 0.18% for BIDMC patients, yielding an overall error rate of 0.34%. Within model 2, the estimated error rate for BIDMC was 0.39% and for MGB was 0.03%, yielding an overall error rate of 0.42%. See Supplemental Table S6 for the observed error rates for each cohort group (Pe), and Supplemental Table S7 for a reconstructed cohort occurrence (p).

WLST Clinical Results

Among the 444 patients who underwent WLST, 189 (42.6%) died within 3.0 days of admission. The time from documented admission to discharge-death was used as a proxy for time-to-WLST and was 6.65 days (SD 8.63 days), with a range of 0.02 to 72.76 days. The 25th, 50th, and 75th percentiles of WLST duration were 1.26, 3.94, and 8.52 days, respectively (see Fig. 5).

Figure 5:

Patient admission lengths before WLST. The x-axis represents admission length in days, while the y-axis shows the number of patients that were admitted before passing away from a withdrawal of care.

Discussion

Principal Findings

Our study shows that combining structured ICD codes with NLP-extracted keywords yields high accuracy on internal holdout data and provides a practical pipeline for scalable phenotyping. However, cross-site variability, particularly for WLST detection, limits generalizability. By automating what has traditionally required labor‑intensive chart review, these models correctly identified patients with cardiac arrest, and those who then died of WLST. Beyond cohort assembly, this model opens the door to granular analyses of neuroprognostication practices, facilitating investigations into whether WLST timing and language contribute to self‑fulfilling prophecies. Moving forward, embedding our pipeline into multicenter EHR platforms, and validating it in larger, more diverse cohorts will be critical to ensure robustness and to inform evidence‑based end‑of‑life quality improve efforts.

Our study demonstrates that combining keyword and ICD code-based features improves EHR phenotyping performance, enhancing our ability to perform retroactive analysis studies. We also found that models based solely on keywords were extremely effective. Adding ICD codes improved classification by only a single additional correct note out of 1,500, suggesting that free-text features alone may be sufficient for phenotyping tasks of this nature.

When examining important features, the model prioritized terms describing the direct actions and status changes associated with WLST (e.g., ‘CMO’, ‘expired’) over phrases describing the rationale (e.g., ‘poor prognosis’), suggesting that documentation of care transitions provides a stronger predictive signal.

Both logistic regression and random forest models performed well, with random forest models performing slightly better. Random forest models often outperform logistic regression in EHR phenotyping tasks due to their ability to capture nonlinear relationships.¹⁸ Given the intended use (cohort assembly for retrospective research), we prioritized high specificity to avoid false positive inclusion; users seeking different trade-offs should recalibrate thresholds to their application.

Clinical Implications

Our by-hand annotation of the entire cohort showed that 47.3% of patients who were diagnosed with cardiac arrest and coma died of WLST, with nearly 42% of those patients receiving WLST within less than 72 hours. Current guidelines recommend against WLST before 72 hours have passed, and the high percentage of WLST within 72 hours suggests that crucial prognostic judgments are being made within this early timeframe. This is a critical window where more objective and accurate prognostic tools could help ensure decisions align with guidelines and prevent premature WLST in patients with recovery potential.¹⁹ Withdrawing life support directly results in the death of the patient, and may represent a self-fulfilling prophecy in many cases. Importantly, these models are not ready for clinical decision support; they are intended to identify retrospective cohorts for research into practice patterns and ethics. However, using this pipeline to study accurate early prognostication could potentially save the lives of patients who have potential for delayed neurological recovery.

Comparison to Prior Research

HR-based phenotyping has been used to classify acute neurological injury patterns. Unsupervised clustering of post–cardiac-arrest data delineated early brain injury subtypes linked to outcomes in over 2,000 patients, and ICD-based algorithms have achieved >90% precision for identifying out-of-hospital arrests in administrative data.^19,20 Our model advances these efforts by automating phenotyping of cardiac arrest, coma, and WLST, enabling large-scale cohort generation without manual review.

Combining natural language processing (NLP) of clinical notes with structured codes consistently outperforms code-only models. Across 172 ICU phenotypes, text features exceeded structured data in 84% of cases, and disease-specific studies have shown 15–30% gains in precision when both are integrated.^9,10 Our hybrid approach, merging ICD codes with curated NLP keywords, achieved AUROCs >0.97, demonstrating high-fidelity phenotyping at scale.

Manual chart review remains the gold standard for identifying WLST but lacks scalability. In one multicenter cohort, ~25% of in-hospital deaths followed WLST for neurologic reasons.21 Recent NLP systems have detected code-status discussions (e.g., DNR/DNI, comfort care) with F1 scores >0.90, matching expert review.²² Our model builds on these advances by automatically identifying WLST events with similar accuracy and minimal manual input.

Limitations & Future Direction

Limiting data to two Boston hospitals restricted generalizability, necessitating broader validation to capture regional linguistic variation. Our 500-word threshold excluded ~80% of notes, limiting applicability to detailed summaries. Future work should adapt models to briefer documentation. Critically, the rigorous nested validation produced small training sets (e.g., n=94); the resulting sensitivity drop highlights that this methodology requires substantial training data to capture linguistic variety for robust cross-hospital generalizability. This likely explains the reduced sensitivity observed when Model 2 was externally applied across institutions. Additional limitations include the use of admission-to-death intervals as proxies for WLST timing, potential error propagation from Model 1 to Model 2 due to training on positive cases from Model 1, and the need for periodic retraining as terminology evolves. Finally, future research must distinguish direct causes of WLST to better understand the influence of prognostication on withdrawal decisions.

Conclusion

In summary, our NLP‑augmented phenotyping framework transforms EHR data into a rapid, reliable means of identifying patients with cardiac arrest, coma, and WLST. By reducing manual abstraction while retaining high accuracy, this approach empowers large‑scale retrospective studies of neuroprognostication and end‑of‑life decision‑making. Future work will leverage these phenotypes to quantify practice variation, examine the impact of documentation language on WLST timing, and ultimately guide the development of standardized, ethically informed prognostic tools.

Figure 2:

Feature importances averaged across cross-validation folds used during training; final importance values were inspected on the single final model. The x-axis represents the average coefficient value of each feature, while the y-axis lists the most influential features. Higher values indicate greater impact on model predictions. Values were determined using Gini impurity.

Data and Code Availability

The code and data used in this study will be made publicly available at the time of publication through the Brain Data Science Platform ( https://bdsp.io/). The code for this project is available at https://github.com/bdsp-core/NAX-CardiacArrest