Natural Language Processing for Risk Stratification of Emergency Department Patients with Chest Pain

Abstract

Background

Natural language processing can transform unstructured clinical data into actionable insights that optimize workflows, improve patient outcomes, and reduce bias. We created a machine learning model that predicted suspicion of acute coronary syndrome (ACS) based on physician notes and assessed its capability to predict major adverse cardiovascular events (MACE) when implemented within the HEART (history, electrocardiogram, age, risk factors, troponin) score.

Methods

We retrospectively reviewed adults presenting with chest pain to The Ottawa Hospital emergency departments (EDs). GPT-4 produced and extracted 11 binary symptom features from physician-documented history of present illness. An extreme gradient boosting (XGBoost) classifier was trained on these features to predict ED disposition, a surrogate marker of physician suspicion of ACS (pACS). The resulting natural language processing–derived ACS suspicion score (nACS) replaced the history component of HEART (nHEART) and was compared with the disposition-based composite (pHEART).

Results

Of 1011 patients (48.5% female; mean age, 55.7 years), 69 (6.8%) experienced MACE within 6 weeks. nHEART achieved sensitivity of 97.1% and specificity of 53.8%; pHEART achieved sensitivity and specificity of 95.7% and 69.1%, respectively. pACS and troponin were the strongest multivariable predictors. GPT-4o extraction showed Fleiss’ kappa > 0.8 across runs and Cohen’s kappa > 0.6 vs extraction by research assistants.

Conclusions

Large language models can extract structured symptom features from ED notes, and an XGBoost model trained on these features can reproduce a coarse measure of physician suspicion of ACS. Clinical deployment will require further model training and testing against a prospectively documented measure of physician ACS suspicion.

Chest pain is one of the most common presentations in the emergency department (ED), accounting for approximately 400,000 visits in Canada annually.¹ Early differentiation of the etiology of chest pain in acute care settings is essential for diagnosis and guiding timely interventions.² Risk stratification scores, such as the HEART (history, electrocardiogram (ECG), age, risk factors, troponin) score, were developed in 2008 to identify the risk of major adverse cardiac events (MACE), including acute myocardial infarction (AMI) within 6 weeks.^3,4 Values of the HEART score range from 0-10 and are based on the patient’s history, age, cardiac risk factors, 12-lead ECG, and cardiac biomarkers during the initial ED visit.^3,4 Patients with HEART scores ≤ 3 are considered low risk and can be discharged home safely, thereby reducing unnecessary additional investigations or hospitalization.⁵ The HEART score has been tested in various retrospective and prospective studies. The score has proven to be a valuable clinical tool in risk stratification of patients presenting to the ED with chest pain.^6,7 However, criticism has been raised regarding the subjectivity of the history portion of the score, with studies reporting discordance in HEART scores among physicians of varying specialties and experience.8, 9, 10

Natural language processing (NLP) software, such as ChatGPT, based on GPT-4 architecture, have garnered significant attention for their potential applications, especially within the field of diagnostic reasoning and objective risk stratification.^11,12 GPT-4 has demonstrated the ability to extract necessary information and apply it in the appropriate clinical context.^13,14 In regard to cardiovascular diagnostic proficiency, GPT-4’s performance has proven to be comparable to that of established prediction models, such as the Framingham and the American Heart Association (AHA) risk scores, and it often outperforms resident physicians in diagnostic accuracy.^15,16

However, concerns have been raised regarding GPT-4’s reliability and reproducibility, with studies reporting severe inconsistencies in acute coronary syndrome (ACS) diagnosis and management when presented with identical patient data.^17,18 It is evident that GPT-4 provides inconsistent and unreliable recommendations when presented with large amounts of unorganized data.

In this study, we combine large language models (GPT-4 for feature discovery and GPT-4o for per-patient feature extraction) with gradient-boosting machine learning to build a structured-feature predictive model from physician-documented notes in real ED visits. Our objective is to test whether such a model can reproduce a coarse measure of physician suspicion for ACS for chest-pain patients and predict 6-week MACE when used in place of the history component of the HEART score.

Methods

Study design

We conducted a health records review of adults presenting to 2 academic EDs within The Ottawa Hospital between November 27, 2019 and January 8, 2020 with a triage label of chest pain with cardiac features. These labels were assigned by triage nurses based on the classic definition of anginal chest pain from the Canadian Cardiovascular Society. Patients were excluded for any of the following reasons: a diagnosis of ST-elevation myocardial infarction (STEMI), age less than 18 years, traumatic chest pain, transfer from another hospital, missing ECG, prior investigation for chest pain within the preceding month, leaving the ED before being seen by a physician, residence outside the Ottawa catchment area, or cardiac arrest in the ED. Patients who did not have chest pain documented on physician chart review, despite a triage label of chest pain with cardiac features, also were excluded. From each retained visit, we collected the patient’s past medication history, the physician’s clinical documentation, laboratory results, ECG, and patient disposition. Our primary outcomes were MACE within 6 weeks of the index ED visit. MACE was defined as acute myocardial infarction, revascularization, hospitalization for unstable angina, and all-cause mortality. Outcomes were ascertained by chart review of all subsequent encounters within The Ottawa Hospital network and were adjudicated independently by 2 reviewers. The physician’s clinical documentation was edited by research assistants (RAs) to isolate the history of present illness (HPI) and remove any patient-identifying information.

A total of 1538 patients with a triage label of chest pain with cardiac features were screened across the 2 campuses. A total of 496 patients were excluded at screening before a study identifier (ID) was assigned:—160 because chart review showed that they did not actually present with chest pain, 136 because they resided outside the Ottawa catchment area, 75 because they left the ED before being seen by a physician, 51 because they had been investigated for chest pain within the preceding month, 50 because of traumatic chest pain, 31 because the visit was a duplicate of a previously enrolled index visit, 7 because of a diagnosis of STEMI, and 1 because of transfer from another hospital. Of the 1042 patients given study IDs, an additional 31 were excluded after chart review—10 because the ECG was missing from the available record, 7 because they had been investigated for chest pain within the preceding month (identified only on chart review), 5 because chart review showed that they did not actually present with chest pain, 4 because they resided outside the Ottawa catchment area, 2 because of traumatic chest pain, 1 because they left the ED before being seen by a physician, 1 because they suffered a cardiac arrest in the ED, and 1 for other reasons. The final analytic cohort consisted of 1011 patients.

Utilization of GPT-4 and GPT-4o

Two distinct tasks were performed using OpenAI large language models. Because these tasks were performed at different time points and with different technical requirements, they used different models and different access modes. Both are reported here explicitly. The verbatim prompts and full application programing interface (API) parameters are provided in Supplemental Appendix S1.

Feature Identification and selection (April 11, 2024, GPT-4 via ChatGPT Web interface)

To identify candidate clinical features predictive of cardiac outcomes, the deidentified HPIs of patients in the high-risk disposition group (n = 123; those who received an inpatient cardiology consultation in the ED) were pasted into a single ChatGPT conversation on April 11, 2024, using a ChatGPT Plus subscription. The exact GPT-4 model snapshot served by the ChatGPT Web interface on that date is not exposed to the user, and temperature, top-p, and seed are not user-configurable through this interface. We therefore report the calendar date as the most specific version information available. Within the conversation, the model was asked to enumerate recurring clinical features across the submitted HPIs. Because all HPIs were included in a single conversation, the output reflects the model’s joint analysis of the group rather than independent per-patient assessments. The model returned 15 candidate features. Four of these overlapped with items already captured by the HEART risk-factor components and were removed to avoid double-counting when the NLP-derived ACS suspicion score (nACS) would later be combined with the HEART risk-factor score; the remaining 11 were retained as the input feature set used throughout the study: chest pain with exertion, central or left-sided chest pain, shortness of breath, diaphoresis, nausea or vomiting, dizziness or presyncope, use of nitroglycerin for relief, exertional dyspnea, pleuritic chest pain, orthopnea or paroxysmal nocturnal dyspnea, and palpitations.

Per-patient feature extraction (October 2024 and May 2025, OpenAI API)

Per-patient feature extraction was performed through the OpenAI Python SDK (not the ChatGPT Web interface) using the pinned model snapshot gpt-4o-2024-05-13. For each patient and each of the 11 features, an independent API call was issued, with no conversation history shared across calls; this single-task per-call structure was chosen to minimize task interference, consistent with prior reports that combining or chaining multiple tasks within a single prompt can degrade large-language-model accuracy.19, 20, 21 Each call used a system message (“You are a heart surgeon designed to output JavaScript Object Notation [JSON].”) and a user message of the following form:

“Read the following patient history. If the following symptom exists, return 1; otherwise, return 0. Symptom: “ x ”. HPI: “y”,

where “ x ” was the literal feature name (for example, “Chest pain with exertion”) and “y” was the full deidentified HPI. The only API parameters explicitly set were the model snapshot, the system message, the user message, and the response format (set to JSON object, to enforce structured output). All sampling and generation parameters, including temperature, top-p, presence penalty, frequency penalty, maximum tokens, number of completions, and seed, were left at the OpenAI API defaults (temperature of 1.0, top-p of 1.0, presence penalty of 0, frequency penalty of 0, and no fixed seed). As a result, each call returned a sample from the model’s output distribution, and run-to-run variation was expected and characterized.

The full feature-extraction procedure was performed a total of 5 times, once in October 2024 and 4 additional times in May 2025, to characterize run-to-run stability. All 5 runs used the pinned snapshot gpt-4o-2024-05-13. Observed variation across runs (Fig. 1) is therefore consistent with stochastic sampling under the default API parameters rather than with changes in the underlying model version. We report both the full 5-run agreement and the agreement restricted to the 4 May 2025 runs, and we use the May 2025 runs as the reference for comparison against manual extraction by RAs.

Figure 1.

Assessing reliability of GPT-4o (gpt-4o-2024-05-13) feature extraction across 5 application programming interface (API) runs.

The extraction capabilities of GPT-4o were then compared to single-pass manual extraction by RAs for 6 of the 11 clinical features: pleuritic chest pain, shortness of breath, nausea or vomiting, diaphoresis, use of nitroglycerin for relief, and chest pain with exertion. The remaining 5 features were not manually extracted due to resource constraints.

Model training and evaluation

We developed an extreme gradient boosting (XGBoost)²² classifier to risk-stratify the history component of the HEART score using the 11 GPT-4o-extracted binary symptom features as the only inputs. The remaining HEART components (age, risk factors, ECG, and troponin) were not used as XGBoost inputs; they entered the analysis only downstream, when the XGBoost-derived nACS was added to them to form the composite nHEART score. The training target was a binary variable derived from the physician’s final disposition: 0 for family physician follow-up (disposition = 1; n = 792), and 1 for ED cardiology consultation (disposition = 2; n = 123). Patients in the intermediate “outpatient cardiology referral” group (disposition categories 3 and 4; n = 96) were excluded from model training because they did not cleanly map to either the low-risk or the high-risk binary label. These 96 patients were retained in the downstream full-cohort evaluation in Table 3, in which the 3-level nACS output was applied to all 1011 patients.

Table 3.

Sensitivity and specificity of physician suspicion of acute coronary syndrome (pACS), natural language processing–derived acute coronary syndrome suspicion score (nACS), pACS within HEART score (pHEART), and nACS within HEART score (nHEART)

	pACS		nACS		pHEART		nHEART
Model risk value	Low	High	Low	High	Low	High	Low	High
No MACE	788	154	325	617	651	291	507	435
MACE within 6 weeks	4	65	4	65	3	66	2	67
Sensitivity	0.94 (0.86–0.98)		0.94 (0.86–0.98)		0.96 (0.88–0.99)		0.97 (0.90–0.99)
Specificity	0.84 (0.81–0.86)		0.35 (0.32–0.38)		0.69 (0.66–0.72)		0.54 (0.51–0.57)
PPV	0.30 (0.24–0.36)		0.10 (0.08–0.12)		0.18 (0.15–0.23)		0.13 (0.11–0.17)
NPV	0.99 (0.99–1.00)		0.99 (0.97–1.00)		1.00 (0.99–1.00)		1.00 (0.99–1.00)
Positive LR	5.76		1.44		3.10		2.10
Negative LR	0.07		0.17		0.06		0.05
AUC (95% CI)	0.93 (0.90–0.97)		0.70 (0.65–0.74)		0.82 (0.80–0.85)		0.75 (0.73–0.78)
Accuracy	0.84		0.39		0.71		0.57
F1 score	0.45		0.17		0.31		0.23
Youden’s J	0.78		0.29		0.65		0.51
Diagnostic odds ratio	83.15		8.56		49.22		39.04

AUC, area under the curve; CI, confidence interval; LR, likelihood ratio; MACE, major adverse cardiovascular events; NPV, negative predictive value; PPV, positive predictive value.

For model evaluation, patients were split into training (75%) and held-out test (25%) sets with stratification on the binary outcome. Given the inherent imbalance in the dataset toward the low-risk class, stratified undersampling of the majority class was employed inside the training fold only, after the train–test split, to produce a balanced training set without introducing information leakage into the test set.

Hyperparameter optimization and performance estimation

Hyperparameter optimization was performed using GridSearchCV with 5-fold cross-validation on the undersampled training set. The parameter grid included learning rate (0.3, 0.1, 0.05), maximum depth (3, 5, 6, 7), and number of estimators (100, 200, 300), yielding 36 candidate hyperparameter combinations. To characterize sampling variability, the entire train–test split and grid-search procedure was repeated across multiple random seeds, and the distribution of test-set areas under the curve (AUCs; Supplemental Fig. S1) and representative receiver operating characteristic (ROC) curves at the highest, median, and lowest AUC seeds (Supplemental Fig. S2) were reported. The model with the median ROC AUC was selected as the most representative model for reporting purposes.

To better characterize out-of-sample performance, we additionally performed nested 5 × 5 cross-validation, consisting of a 5-fold outer loop with a 5-fold inner grid search, repeated across 5 outer seeds for a total of 25 outer-fold evaluations, with undersampling applied inside each outer-training fold only. The nested-CV AUC was 0.783 ± 0.061 (mean ± standard deviation; 95% confidence interval [CI] 0.759-0.807; per-fold range, 0.671-0.868), consistent with the single-split median of 0.776. The held-out test set AUC for the representative single model was 0.79 (95% CI 0.69-0.89 by DeLong’s method,²³ n = 230). Per-fold results and selected hyperparameters are reported in Supplemental Table S1.

Risk stratification and threshold selection

The XGBoost model produced a continuous predicted probability of ACS-related workup. Thresholds for discretizing this output were placed at the local minima of the kernel-density estimate of the predicted probabilities on the full cohort (Supplemental Fig. S3): 0.286 (separating a low-probability cluster centred near 0.07-0.18 from an intermediate cluster centred near 0.32) and 0.412 (separating the intermediate cluster from a broad high-probability tail). Three-level nACS matches the 0/1/2 structure of the HEART history component (≤ 0.286 → 0; 0.286-0.412 → 1; > 0.412 → 2). For the binary sensitivity/specificity analysis in Table 3, the lower cutoff (0.286) was used. The same binary logic was applied to the physician-based suspicion (pACS). pACS = 0 (family physician follow-up) was treated as low-risk; pACS = 1 or 2 (outpatient cardiology referral or ED cardiology consultation) was treated as high-risk. For the composite physician HEART score (pHEART) and NLP model HEART score (nHEART scores), the established HEART cutoff of ≤ 3 for low-risk and ≥ 4 for high-risk was used.

A 3-level comparison is reported in Supplemental Table S2. Calibration of the XGBoost probabilities, together with Platt-scaled and isotonic regression recalibration variants, is reported in Supplemental Figure S4 and Supplemental Table S3.

Physician suspicion of ACS (pACS) vs NLP model output (nACS)

Although a structured clinical impression field exists within our dataset, we could not guarantee standard practices across all our staff and resident physicians. Therefore, we decided to use ED disposition as a surrogate measure for suspicion of ACS, termed pACS. The pACS variable was coded as 0 (low-risk) if the patient was discharged home with family physician follow-up, 1 (intermediate-risk) if discharged home with an outpatient cardiology referral, or 2 (high-risk) if a cardiology consult was obtained in the ED.

pACS is not a pure measure of physician gestalt. ED disposition is itself a composite decision that reflects the full HEART score workup (history, ECG, troponin, age, and risk factors) as well as patient factors. When pACS is included in a multivariable model of MACE, it captures prognostic information from the other HEART components, attenuating their independent coefficients and inflating its own. Comparisons of pACS with nACS should therefore be interpreted as comparisons of the NLP model’s output to the patient’s final ED disposition, rather than to an independent measure of physician-only reasoning. We discuss this further in the limitations section.

The nACS was generated by providing the trained XGBoost model with the 11 GPT-4o extracted binary features from each patient’s HPI. nACS was coded on the same 0/1/2 scale. For each patient, the pACS was then compared against the nACS. Per-patient pACS and nACS scores are compared in Table 3; McNemar’s test results are reported in Table 4, and both inclusion and exclusion of pACS from the multivariable model are reported in Supplemental Table S4.

Table 4.

McNemar’s test analysis for sensitivity and specificity of natural language processing–derived acute coronary syndrome suspicion score (nACS) vs physician suspicion of acute coronary syndrome score (pACS), nACS within HEART score (nHEART) vs pACS within HEART score (pHEART)

		nACS correct	nACS incorrect		nHEART correct	nHEART incorrect
McNemar’s test for sensitivity—positive MACE analysis	pACS correct	62	3	pHEART correct	64	2
	pACS incorrect	3	1	pHEART incorrect	3	0
	P = 0.68, no statistical significance between sensitivities			P = 1.00, no statistical significance between sensitivities

		nACS correct	nACS incorrect		nHEART correct	nHEART incorrect
McNemar’s test for specificity—negative MACE analysis	pACS correct	308	480	pHEART correct	505	146
	pACS incorrect	17	137	pHEART incorrect	2	289
	P < 0.0001, pACS is statistically more specific			P < 0.0001, pHEART is statistically more specific

MACE, major adverse cardiovascular events.

Incorporation of the physician and NLP models for ACS suspicion within the HEART score

For each patient, the following components of the HEART score, including age, risk factors, ECG, and troponin values, were manually extracted by RAs and stored in an online Webform.

For each variable, a score of 0, 1, or 2 was assigned based on the HEART score criteria.

ECGs were interpreted by a staff emergency medicine physician during data collection—0 if normal, 1 if nonspecific repolarization changes as well as left bundle branch block and left ventricular hypertrophy, 2 if significant ST segment deviation.

Age was assigned as follows: 0 if aged < 45 years, 1 if aged 45-65 years, and 2 if aged > 65 years.

Risk factors were assigned as follows: 0 if no risk factors, 1 if 1-2 risk factors, 2 if > 3 risk factors or history of atherosclerotic disease. Risk factors included hypertension, hypercholesterolemia, diabetes, obesity with body mass index > 30 kg/m², smoking, and positive family history. Atherosclerotic disease included prior myocardial infarction, coronary artery bypass grafting/ percutaneous coronary intervention, cerebrovascular accident, or transient ischemic attack.

Troponin was assigned as follows: 0 if within normal limits, 1 if 1-3 times the normal limit, 2 if > 3 times the normal limit. The Ottawa Hospital uses the Roche high-sensitivity cardiac troponin T (hs-cTnT) assay; a value of ≤ 14 ng/L is considered within normal limits.

The pACS and nACS scores were added to the ECG, age, risk factors, and troponin scores to create the pHEART and the nHEART.

Statistical analysis

All analyses were performed in Python using pandas, scikit-learn, XGBoost, statsmodels, and scipy. Univariate and multivariable logistic regression were used to estimate odds ratios for each HEART component and MACE. Inter-rater agreement across GPT-4o runs was quantified using Fleiss’ kappa (for 3 or more raters) and Cohen’s kappa (for pairwise GPT-4o vs RA comparisons). Paired comparisons of classifier sensitivities and specificities were performed using McNemar’s test. Model discrimination was assessed by the ROC AUC, with 95% Cis computed using DeLong’s method.²² Cis for sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were computed using the Wilson score method. Model calibration was assessed using the Brier score, a decile-based calibration plot, and the Hosmer-Lemeshow goodness-of-fit test.²⁴ Because training used random undersampling of the majority class, the raw XGBoost probabilities are biased relative to the population prevalence. We therefore also report 2 recalibrated variants, Platt scaling (a logistic wrapper),²⁵ and isotonic regression,²⁶ each fit on 5-fold out-of-fold predictions drawn from the training set only, so that the held-out test set remained untouched by the calibrator. All statistical tests were 2-sided with an alpha of 0.05.

Results

As shown In Table 1, 1011 patients were enrolled in the study. A total of 123 patients received a cardiology consultation in the ED; 96 patients were referred for outpatient cardiology follow-up; and 792 patients were discharged home with no cardiology follow-up.

Table 1.

Demographics of patients based on emergency department (ED) disposition

Characteristic	Disposition from ED, n (%) or mean ± SD
	ED cardio consultation (n = 123)	Outpatient referral (n = 96)	Family physician follow-up (n = 792)
Demographic
Female	39 (31.7)	43 (44.8)	408 (51.5)
Age, Y	69.5 ± 14.1	60.5 ± 12.8	53.0 ± 18.2
Past medical history
CAD	70 (56.9)	30 (31.3)	143 (18.1)
Family history of CAD	17 (13.8)	16 (16.7)	54 (6.8)
Hypertension	92 (74.8)	42 (43.8)	265 (33.5)
Hypercholesterolemia	72 (58.5)	45 (46.9)	184 (23.2)
Diabetes mellitus	39 (31.7)	21 (21.9)	108 (13.6)
Obesity (BMI > 30 kg/m2)	36 (29.3)	23 (24.0)	117 (14.8)
Current smoker	22 (17.9)	11 (11.5)	106 (13.4)
Cerebrovascular accident	11 (8.9)	4 (4.2)	29 (3.7)
ED visit
Positive troponin	89 (72.4)	17 (17.7)	110 (13.9)
Abnormal ECG	41 (33.3)	21 (21.9)	109 (13.8)
Major adverse cardiac events	63 (51.2)	2 (2.1)	4 (0.5)

BMI, body mass index; CAD, coronary artery disease; ECG, electrocardiogram; SD, standard deviation.

A total of 49 patients (4.8% of the cohort) did not have a high-sensitivity troponin drawn during their index ED visit. To keep the patient population consistent with our true ED population and to give our model more examples of extremely low-risk chest pain presentations, the decision was made to keep these patients within our analysis and assume a negative troponin value for them, rather than exclude them. A sensitivity analysis restricted to the 962 patients with measured troponin is reported in Supplemental Table S5; the headline metrics for nACS and pACS change by 0.011 or less for each of sensitivity, specificity, PPV, NPV, and AUC, and the conclusions of Tables 3 and 4 are unchanged in direction and statistical significance.

Patients who received a cardiology consult in the ED were more likely to be male, older, and have higher rates of cardiac risk factors than patients who were discharged home or received an outpatient cardiology referral. In addition, patients who received a cardiology consultation in the ED had higher incidences of abnormal troponin values and an abnormal ECG.

A total of 69 patients (48 male, average age 68.0 years; 21 female, average age 72.8 years) had a MACE within 6 weeks of ED presentation—55 with cardiovascular revascularizations, 8 with non-ST-elevation myocardial infarction, 4 with unstable angina, and 2 deaths.

Of the 69 patients who had a MACE within 6 weeks, cardiology was consulted in the ED for 63 patients, 2 were referred to cardiology as an outpatient, and 4 patients were sent home to follow up with their family physician. A total of 63 patients had a MACE during the initial visit, and 6 patients had a MACE within the next 6 weeks.

Within Table 2, we utilized the pACS score based on patient ED disposition and assessed how each subcomponent of the HEART score was associated with MACE within 6 weeks of the index ED visit. All 5 subcategories had odds ratios above 1 and P-values < 0.001 in univariate analysis. However, when multivariable logistic regression was performed, only pACS and the troponin subcategories had significant P-values. We assessed the importance of each model through multiple metrics, including mean decrease Gini and the Least Absolute Shrinkage and Selection Operator (LASSO) coefficient. The mean decrease Gini represents the total reduction in the impurity index, whereas the LASSO coefficient represents the estimated impact of a variable. In both metrics, the higher the number, the greater the importance of the variable in determining the outcome. From our analysis, pACS clearly was by far the most important factor in predicting MACE within 6 weeks. The second most important factor was the troponin value, with all other categories having significantly less importance.

Table 2.

Assessing the correlation of each component of the HEART (history, electrocardiogram [ECG], age, risk factors, troponin) score with positive major adverse cardiovascular events (MACE) within 6 weeks of index emergency department visit (n = 1011; 49 missing-troponin cases imputed as 0)

HEART component	pACS	Troponin	ECG	Age	Risk factor
Univariate logistic regression odds ratio (95% CI)	17.89 (10.20–31.41)	7.31 (5.16–10.34)	2.80 (2.00–3.92)	2.88 (1.96–4.23)	3.05 (2.08–4.48)
Univariate logistic regression P-value	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001
Multivariable logistic regression odds ratio	12.23	2.85	1.18	0.67	0.95
Multivariable logistic regression P-value	< 0.001	< 0.001	0.48	0.22	0.85
Mean decrease Gini	52.04	23.15	9.19	7.49	8.14
LASSO coefficients	2.32	0.98	0.12	–0.32	–0.03
AUC	0.93	0.84	0.61	0.69	0.71

AUC, area under the curve; CI, confidence interval; LASSO, least absolute shrinkage and selection operator; pACS, physician suspicion of acute coronary syndrome.

We assessed the reliability and reproducibility of GPT-4o when extracting the presence or absence of each clinical feature in Figure 1. Across all 5 runs, for the 11 clinical variables, GPT-4o demonstrates near-perfect agreement, with a Fleiss’ kappa > 0.8 for all clinical features. The feature with the highest agreement is nausea/vomiting, and the feature with the lowest agreement is central/left-sided chest pain. The Fleiss’ kappa for all variables improves when the last 4 runs alone are analyzed.

We analyzed the performance of GPT-4o in feature extraction compared to medical RAs in Figure 2. Due to resource constraints, RAs completed single-pass manual data extraction of only 6 of the 11 variables—pleuritic chest pain, shortness of breath, nausea or vomiting, diaphoresis, relief of pain with nitroglycerin, and chest pain with exertion. The RA extraction was compared to the average of the last 4 GPT-4o runs in May 2025. Across all 6 variables, GPT-4o achieved an accuracy greater than 90% compared to RAs. Four of the 6 variables had near-perfect agreement, with a Cohen’s kappa > 0.8, and the other 2 variables had substantial agreement, with a Cohen’s kappa > 0.6. The feature with the highest agreement is pleuritic chest pain, and the feature with the lowest agreement is chest pain with exertion.

Figure 2.

Assessing feature extraction of GPT-4o (gpt-4o-2024-05-13) in the 4 May 2025 API runs vs manual extraction by research assistants.

Table 3 summarizes the performance of the NLP-based and physician-based scores. nACS matched pACS on sensitivity (both 94.2%), but trailed on specificity (34.5% vs 83.7%), corresponding to 617 false positives for nACS vs 154 for pACS. Combining each score with the remaining HEART components partially closed the gap: nHEART achieved slightly higher sensitivity than pHEART (97.1% vs 95.7%) and modestly lower specificity (53.8% vs 69.1%). The 4 non-history HEART components therefore offset much of the over-triage observed when nACS is used alone.

As shown in Table 4, when comparing the performance of nACS vs pACS and nHEART vs pHEART using McNemar’s test, which examines the differences in classification between models, we find no statistical significance in the sensitivities. However, the physician-based models, pACS and pHEART, are significantly more specific than the NLP model.

Discussion

nACS achieved sensitivity comparable to that of pACS for 6-week MACE (94.2% vs 94.2%; McNemar’s P = 0.68) but at substantially lower specificity (34.5% vs 83.7%; P < 0.0001). Combining either score with the remaining HEART components partially closed this gap: nHEART approached pHEART on sensitivity (97.1% vs 95.7%; P = 1.0) and trailed on specificity (53.8% vs 69.1%; P < 0.0001). These findings suggest that NLP-derived history features can preserve the sensitivity of a HEART-based screen while flagging more patients for physician review. This trade-off may be acceptable in low-risk rule-out applications, but it is not adequate in its current form for autonomous triage.

The relative importance of the HEART components shifted markedly between univariate and multivariable analysis. All 5 components were significant in univariate analysis, but only pACS and troponin retained significance under multivariable logistic regression. This pattern suggests that a weighted combination of HEART components, of the type produced by machine learning algorithms, may discriminate MACE more efficiently than the score’s original equal-weight additive structure. Any such weighting would need to be re-derived and validated for each deployment site. The collapse of age, risk factors, and ECG to nonsignificance in the multivariable model likely reflects 2 distinct phenomena. First, as discussed above, the disposition-based pACS captures information from all 5 HEART components, so the multivariable coefficients on the other components are attenuated as a consequence. Second, even after removing pACS from the multivariable model (Supplemental Table S4), troponin remained the dominant predictor, ECG and risk-factor count recovered statistical significance (P = 0.023 and P = 0.038 respectively), and age remained nonsignificant.

GPT-4o feature extraction was reliable both within itself and as compared to human raters: across the 4 May 2025 runs, Fleiss’ kappa exceeded 0.8 for every clinical feature, and pairwise Cohen’s kappa, against single-pass extraction by RAs, exceeded 0.6 for all 6 dual-extracted features. Run-to-run variation was small and is consistent with stochastic sampling under the default API parameters rather than with changes in the underlying model, as all 5 runs used the same pinned snapshot. Given that single-pass manual extraction by RAs itself carries inherent error, these findings support the feasibility of using NLP to expedite structured-feature extraction from clinical text.

Prior reports have documented poor reproducibility when GPT-4 is asked to produce free-form clinical recommendations from large, unstructured inputs, for example in chest pain risk stratification and ED triage.^17,18 By contrast, when we constrained GPT-4o to a single binary task per call (extracting one symptom from one patient’s history), agreement was high both within itself (Fleiss’ kappa > 0.8 across runs) and against human raters (Cohen’s kappa > 0.6 in all dual-extracted features). Large language models are not yet reliable as autonomous clinical decision-makers, but with sufficiently constrained prompts, they can produce structured features that downstream machine-learning models can use.

Although the sensitivity of nHEART (97.1%) is attractive, the low specificity of nACS (34.5%) and nHEART (53.8%) is not adequate for immediate operational deployment as a triage tool. In our cohort, nACS would flag 617 of 942 non-MACE patients as high-risk. The high negative predictive value (0.99 for nACS, 1.00 for nHEART; Table 3) suggests that the most plausible utility is in identifying very-low-risk presentations. The raw XGBoost probabilities are biased by undersampling during training; Platt or isotonic recalibration on training-set out-of-fold predictions restored calibration (Brier 0.09 vs 0.18 raw; Hosmer-Lemeshow P = 0.18 and 0.56; calibrated probabilities reported in Supplemental Table S3). External validation in a larger, more diverse cohort would be required before deployment.

This study is the only one we are aware of that assesses the accuracy and reliability of GPT-4o in extracting tabular clinical data from actual ED physician-documented clinical notes. In addition, we investigated how this information can be used to develop complex risk-stratification algorithms when combined with machine-learning algorithms. Strengths of this study include a moderate sample size, with 69 MACE events, and a robust comparison of the NLP/machine learning model’s performance against the physician-based model.

Limitations

Our study has three principal limitations.

First, the comparator pACS is a proxy for physician suspicion of ACS rather than a pure measure of physician gestalt. ED disposition is a downstream, composite decision that reflects the full HEART workup (history, ECG, troponin, age, and risk factors) as well as other patient variables. In the multivariable model, pACS therefore captures information from the other HEART components, driving age, risk-factor count, and ECG to nonsignificance; Supplemental Table S4 shows that ECG and risk-factor count recover statistical significance once pACS is removed. The apparent specificity advantage of pACS over nACS (84% vs 35%) overstates the true performance of a history-only comparator because pACS carries troponin and ECG information, and the near-perfect AUC of pACS (0.93) for MACE reflects reproduction of the process of care rather than pure prognostic discrimination by physician judgment. The comparison of nACS against pACS is therefore best interpreted as recapitulation of physician workup decisions rather than as an independent comparison of history-taking accuracy. A structured clinical impression field does exist in our dataset, but documentation practices were not standardized across our staff and resident physicians during the study window. Therefore, we could not use it as a reliable independent suspicion-only comparator; this is a key priority for future validation, ideally in a cohort in which physician suspicion is captured under standardized documentation practices at the time of the index ED visit.

Second, single-site sampling variability limits how confidently a single train–test split can characterize the model. We observed substantive variability in test-set AUC across random train–test splits (Supplemental Fig. S1), indicating that certain instances in the dataset are under-represented relative to the predominant distribution. We mitigated this issue with nested 5 × 5 cross-validation (Supplemental Table S1), which yielded an AUC of 0.783 ± 0.061; the single-split estimate fell well within this distribution.

Third, external generalizability has not been established. The model was trained and evaluated within a single Canadian academic centre over a 6-week window. Performance at sites with different documentation conventions, troponin assays, or patient populations cannot be inferred from the present analysis.

This work is a proof of concept that large language models can extract structured symptom features from unstructured ED notes reliably enough to support downstream risk modelling within the HEART score framework. Future work should pursue prospective validation against an independently documented measure of physician ACS suspicion, multi-site evaluation across documentation styles and troponin assays, and dedicated modelling for populations in whom ACS presents atypically, including women and older adults.

Ethics Statement

This study was approved by the Ottawa Health Sciences Network Research Ethics Board. The research reported in this paper adhered to TCPS 2 guidelines.

Patient Consent

The authors confirm that patient consent is not applicable to this article. Patient consent was not required because this study was a retrospective review of existing clinical data. Data collected were de-identified manually by research assistants and therefore, deemed to pose minimal risk to patients.

Funding Sources

G.C. was funded to work on this project through the Mach-Gaensslen Foundation of Canada and the University of Ottawa Summer Research Program.

Disclosures

V.T. reports receiving support through a Physicians’ Services Incorporated Foundation Mid-Career KT fellowship award and a University of Ottawa Tier-1 Clinical Research Chair in Cardiovascular Emergencies award; receiving grant funding from the following peer-reviewed grant funding agencies—Physicians’ Services Incorporated Foundation, Canadian Institutes of Health Research, Heart and Stroke Foundation of Canada, Networks of Centres of Excellence through the Cardiac Arrhythmia Network of Canada (CANet), Ontario Centres of Innovation, and The Ottawa Hospital Academic Medical Organization; and being a consultant for the National Institutes of Health –funded project Practical Approaches to Care in Emergency Syncope (PACES). All the other authors have no conflicts of interest to disclose.