Performance of GPT-5.5 thinking for Manchester Triage System–based emergency department triage: agreement with an expert reference standard and prediction of short-term outcomes

Abstract

Objective

To evaluate the agreement between Manchester Triage System (MTS) category assignments generated by GPT-5.5 Thinking and an expert-derived reference standard in a selected cohort of emergency department (ED) patients, and to assess the model’s ability to predict hospital admission and 72-hour mortality.

Methods

This single-center observational study prospectively sampled consecutive ED presentations during a prespecified continuous 72-hour window, with triage-time variables and secondary outcomes obtained from hospital records. Patients classified as MTS Category 5 (Blue; nonurgent) by the expert reference standard were excluded from the analytic cohort. GPT-5.5 Thinking was prompted in a standardized single-shot format using structured clinical vignettes derived from triage data. The primary endpoint was agreement between model-assigned MTS categories and an expert reference standard established by two independent blinded emergency medicine specialist rater teams, with disagreements resolved through senior adjudication. Exact agreement, within-1-level agreement, overtriage, undertriage, and dangerous undertriage were calculated. Secondary analyses evaluated prediction of hospital admission and 72-hour mortality using diagnostic performance metrics.

Results

A total of 1,841 patients were included. Overall agreement between GPT-5.5 Thinking and the expert reference standard was substantial (κ = 0.721; 95% CI 0.691–0.740). Exact agreement rate was 62.0%, and within-1-level agreement rate was 99.0%. Overtriage occurred in 35.3% of cases, undertriage in 2.7%, and dangerous undertriage in 0.2%. Exact agreement rate was significantly lower in patients aged ≥ 65 years than in younger adults (57.1% vs. 64.2%, p = 0.013). For hospital admission, accuracy rate was 79.9%, sensitivity 83.8%, positive predictive value (PPV) 45.2%, and negative predictive value (NPV) 95.9%. For 72-hour mortality, accuracy rate was 86.5%, sensitivity 72.3%, PPV 12.6%, and NPV 99.1%.

Conclusion

Within this selected ED cohort, GPT-5.5 Thinking showed substantial agreement with expert MTS-based ED triage and generally preserved the ordinal hierarchy of clinical urgency. Its performance was characterized by frequent adjacent-level agreement, predominant overtriage, and very rare but nonzero dangerous undertriage. The model also showed stronger rule-out than rule-in characteristics for hospital admission and 72-hour mortality.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12873-026-01704-7.

Introduction

Emergency departments (EDs) are high-stakes clinical settings in which rapid, accurate decisions directly influence patient outcomes. The global increase in ED visits has contributed to overcrowding, thereby increasing the need to promptly identify patients who require time-sensitive interventions [1]. Triage is a system that aims to prioritize patients according to their clinical urgency, enabling early recognition of life-threatening conditions while supporting appropriate management of lower-acuity presentations. The effective implementation of triage has been associated with a reduction in waiting times, an enhancement in patient safety, and an optimization in the utilization of ED resources [1].

Despite the widespread implementation of conventional triage systems, these systems depend heavily on the clinical judgment of healthcare providers, which renders them susceptible to interrater variability. Such variability may lead to both overtriage and undertriage, with implications for patient safety and resource allocation. In a large-scale evaluation of more than 5 million ED visits in the Kaiser Permanente CREST (Clinical Research on Emergency Services & Treatments) cohort, Sax et al. reported that the Emergency Severity Index (ESI) overestimated acuity in 28.9% of encounters and underestimated acuity in 3.3% [2]. These limitations have prompted interest in artificial intelligence (AI)–enabled triage and, more recently, large language models (LLMs) as decision-support tools. In a systematic review, Ahmed Abdalhalim et al. suggested that AI-based triage systems have the potential to reduce triage time, improve documentation, and enhance decision support in ED workflows [3]. However, a subsequent systematic review by Porto emphasized that the current evidence remains insufficient for routine clinical integration and is limited by heterogeneity, study design constraints, and a paucity of prospective evaluations [4]. Evidence specific to LLMs remains similarly heterogeneous. For instance, Colakca et al. reported a 76.6% agreement for LLM-assisted triage guided by ESI principles [5], whereas Zaboli et al. found limited concordance with the Manchester Triage System (MTS) and lower performance than triage nurses for predicting 72-hour mortality [6]. Collectively, current evidence underscores both the potential and unresolved safety and reliability concerns of LLM-supported triage, particularly when applied to established rule-based systems and when linked to clinically meaningful short-term outcomes.

Therefore, this study evaluated the agreement of triage decisions generated by the GPT-5.5 Thinking model using the MTS in a selected cohort of ED presentations against an expert-derived reference standard across acuity categories. We additionally examined the model’s ability to anticipate clinically relevant short-term outcomes, including 72-hour mortality and hospital admission.

Methods

Ethics committee approval and informed consent

This was a single-center observational diagnostic accuracy study based on prospectively sampled consecutive ED presentations during a prespecified 72-hour window, using routinely collected triage-time data and hospital-recorded outcome data. The study protocol was approved by the Institutional Review Board (IRB) of Haseki Training and Research Hospital, Istanbul, Türkiye (Decision No: 53-2026). The study was conducted in accordance with the ethical principles of the Declaration of Helsinki (2024 revision). Because data and accompanying text do not contain potentially identifiable patient information, the IRB waived the requirement for patient consent for the review of medical records.

Reporting followed the STARD-AI reporting guideline, and LLM-specific methodological details were structured in accordance with the TRIPOD-LLM checklist.

Study design and setting

The study was conducted in the ED of Haseki Training and Research Hospital, a high-volume tertiary referral center in Istanbul with approximately 720,000 annual ED visits. The hospital is a regional tertiary training and research hospital that provides emergency care for both adult and pediatric patients.

Consecutive presentations were sampled during a prespecified 72-hour period from 8:00 a.m. on March 10, 2026, to 8:00 a.m. on March 13, 2026. All patients presenting during this period who met the eligibility criteria were included without discretionary selection.

Data sources, participant selection, and data quality

Primary data were extracted from the hospital’s automated system and ED triage forms were documented contemporaneously during routine ED care by on-duty emergency medicine physicians.

Data collected for each patient encompassed demographic characteristics, vital signs on admission, complaints at admission, a brief clinician-documented history, comorbidities, patient-reported numeric rating scale pain score (0–10), and Glasgow Coma Scale score. Clinical outcomes, including ED disposition (discharge vs. hospital admission) and all-cause 72-hour mortality, were also recorded. These secondary outcome data were extracted from hospital electronic medical records. Hospital admission/discharge status and 72-hour all-cause mortality were not assigned, inferred, or adjudicated by the expert rater teams. For patients with repeat presentations during the sampling period, only the first eligible encounter was included.

To ensure data integrity, a predefined duplicate cross-check procedure was utilized to verify the dataset. This procedure was carried out independently by three emergency medicine residents, each with a minimum of three years of clinical experience.

During the data cleaning process, 82 patients were excluded due to missing key vital parameters or history fields, and 7 patients were excluded due to cardiopulmonary arrest upon arrival. Furthermore, 62 cases were excluded due to restricted-access medical records (e.g., institutional or forensic administrative record locks) that precluded research use. In the final analysis, 2,725 cases classified as nonurgent MTS Category 5 (Blue) by the expert reference standard were excluded from the analytic cohort. The final analytic dataset comprised 1,841 patients.

Manchester triage system

MTS is a widely used five-level ED triage system designed to prioritize patients according to clinical urgency. It categorizes patients into Red (immediate), Orange (very urgent), Yellow (urgent), Green (standard), and Blue (nonurgent) levels based on presenting complaints and predefined clinical discriminators, such as abnormal vital signs, pain severity, bleeding, consciousness level, or risk of deterioration [7].

In the present study, MTS was selected because its complaint-oriented flowcharts and predefined discriminator structure provide a standardized, rule-based, and clinically interpretable framework for evaluating LLM-generated triage decisions. This structure allowed the model’s outputs to be assessed not only as isolated category assignments, but also in terms of ordinal urgency, overtriage, undertriage, and clinically dangerous undertriage.

Reference standard and expert adjudication

The reference standard was established by two independent rater teams of three board-certified emergency medicine specialists. The six specialists each had a minimum of five years of clinical experience. The raters independently reviewed all case forms according to MTS algorithms and were blinded to each other’s decisions and to patient outcomes. To quantify the reliability of the expert reference standard, agreement between the two independent rater teams was calculated before final consensus adjudication. Inter-team agreement was strong for unweighted category assignment (Cohen’s κ = 0.835). Because MTS categories represent an ordinal urgency scale, weighted agreement coefficients were also calculated, showing linear-weighted κ = 0.880 and quadratic-weighted κ = 0.922.

For each case, experts first assigned the most appropriate MTS category according to the five-level MTS framework.

In our institution and in the Turkish emergency care setting, nonurgent presentations constitute a large proportion of ED visits. Because the primary objective was to evaluate model performance among presentations classified as requiring greater urgency than MTS Category 5, patients classified by the expert reference standard as Category 5 (Blue; nonurgent) were excluded from the analytic cohort. This exclusion resulted in a selected analytic population comprising MTS Categories 1–4.

Disagreements between the two rater teams regarding triage category assignment or inclusion/exclusion status were adjudicated by a third senior reviewer, and a final consensus determination was recorded.

Vignette generation and blinding

For each eligible patient, a structured clinical vignette was generated using a predefined standardized template based on routinely collected triage-time data. The source fields included age, sex, presenting complaint, brief clinician-documented triage history, admission vital signs, pain score, Glasgow Coma Scale score, and recorded comorbidities. Raw data were converted into concise text-based vignettes without adding interpretive clinical judgments, diagnostic labels, treatment information, disposition information, or outcome data.

The vignette preparation process was performed by emergency medicine residents who were not involved in the final expert adjudication process. The vignette writers were blinded to the expert reference MTS category, hospital disposition, and 72-hour mortality status.

AI model and prompting strategy

We evaluated the GPT-5.5 “Thinking” model developed by OpenAI (San Francisco, California). All model inferences were run via the OpenAI application programming interface using GPT-5.5 “Thinking” (gpt-5.5-thinking), with constant parameters (temperature = 0, top_p = 1.0), without the utilization of tools or external retrieval, and the API-returned model version/date-stamp documented for each inference (evaluation period: May 29–31, 2026). To reduce the risk of information carryover across cases and potential data leakage, each patient vignette was submitted as an independent request with no prior conversation context.

A structured system prompt, based on a role–task–constraint framework (see Appendix 1), was employed to standardize model behavior and outputs across cases. For each case, single-shot prompting was employed, with a single model response per case, without iterative re-prompting or correction. The model outputs were stored in their original form as raw text and subsequently transformed into a structured dataset for statistical analysis.

The primary analysis used a safety-oriented prompt that instructed the model to assign the higher-acuity category when a case was considered borderline between two categories. To evaluate the influence of this instruction on triage behavior, an additional sensitivity analysis was performed using a neutral prompting framework that omitted the explicit instruction to up-triage borderline cases. All other inference conditions were kept unchanged.

Outcomes and performance measures

The primary outcome was the agreement between the GPT-5.5 Thinking-assigned MTS color category and the expert-derived reference standard. Agreement was evaluated using a patient-safety–oriented ordinal classification comprising three distinct levels: strict success, overtriage, and undertriage. An exact match between the model and the reference standard was considered strict success. The term “overtriage” was defined as the assignment of patients to a higher acuity category. The undertriage of patients was identified as a critical error and a potential safety violation due to the assignment of patients to a lower acuity category. The term “clinically dangerous undertriage” was defined as the assignment of a patient to a level of acuity that is two or more levels lower than the expert consensus.

The secondary endpoint assessed the model’s ability to predict clinically relevant short-term outcomes. We evaluated binary model predictions for hospital admission and 72-hour mortality risk against observed outcomes. Hospital admission was defined as transfer from the ED to an inpatient ward, ICU, operating room, or another facility for inpatient-level care. Patients discharged directly or after an observation stay of less than 24 h without subsequent hospitalization were classified as discharged.

Statistical analysis

Based on prior literature and clinical utility standards, we established a minimum acceptable agreement (null hypothesis, κ₀) of 0.60 (moderate-to-substantial agreement) and anticipated a true agreement (alternative hypothesis, κ₁) of 0.75 for the LLM. To detect this difference with 90% power (1 - β) and a two-sided type I error rate (α) of 0.05, a minimum sample size of approximately 315 patients was required.

However, to account for potential data exclusion (e.g., missing physiological parameters, Category 5 exclusions) and to maintain high statistical power (> 90%) for planned exploratory subgroup analyses (e.g., trauma vs. non-trauma, older adults), we extended the recruitment strategy to a continuous 72-hour sampling window.

Statistical analyses were performed using IBM SPSS Statistics for Windows, version 28.0 (Armonk, NY). Distributional assumptions for continuous variables were assessed using histograms and the Kolmogorov–Smirnov test. Descriptive statistics are presented as counts (n) and percentages (%) for categorical variables and as mean ± standard deviation (SD), and median with interquartile range (IQR) for continuous variables, as appropriate.

Agreement between the GPT-5.5 Thinking model and the reference standard for ordinal MTS categories was assessed using quadratic-weighted Cohen κ, interpreted using the Landis and Koch criteria. To ensure robust precision, 95% confidence intervals (CIs) for κ values were calculated using a bootstrapping method with 1000 iterations. Clinical safety and operational accuracy were further evaluated by calculating proportions for strict success (exact match), adjacent-category agreement (within ± 1 level), overtriage, and undertriage. The 95% CIs for all categorical proportions were computed using the Wilson score interval method.

To evaluate the effect of prompting strategy on model behavior, paired comparisons between the safety-oriented and neutral prompting frameworks were additionally performed. Differences in paired categorical outcomes, including triage category distribution, strict success, overtriage, and undertriage rates, were analyzed using McNemar or Stuart–Maxwell tests, as appropriate. To assess systematic differences in category assignment between the model and the reference standard, we used tests of marginal homogeneity (Bowker’s test of symmetry) as appropriate.

For secondary endpoints, diagnostic performance metrics (sensitivity, specificity, positive predictive value [PPV], and negative predictive value [NPV]) were calculated with 95% CIs to evaluate the model’s ability to predict hospital admission and 72-hour mortality. Performance differences across clinical subgroups (e.g., age distribution, sex, chief complaint category) were evaluated using chi-square (χ²) tests, Fisher exact tests, or Kruskal–Wallis tests; when applicable, post hoc pairwise comparisons were adjusted using the Bonferroni correction.

All tests were 2-sided, and p < 0.05 was considered statistically significant.

Results

Interrater reliability of the expert reference standard

Interrater reliability analysis showed strong agreement between the two rater teams in MTS triage category assignment (unweighted Cohen κ = 0.835). Agreement increased when accounting for the ordinal structure of MTS categories (linear-weighted κ = 0.880; quadratic-weighted κ = 0.922).

Study population

Table 1 summarizes baseline demographic and clinical characteristics of the study population. A total of 1841 patients, including 926 females (50.3%) and 915 males, were included. The median age was 50.0 years (IQR, 33.0–68.0). The most common comorbidities were hypertension (24.5%) and diabetes mellitus (9.6%). Medical (non-surgical) problems were the most frequent chief complaint category (45.3%), followed by neurologic (16.4%) and cardiac (11.3%) problems. Overall, 314 patients (17.1%) were admitted to the hospital and 72-hour all-cause mortality was 2.6% (n = 47).

Table 1.

Baseline demographic and clinical characteristics of the study population (n = 1841)

Characteristic	Value
Age in years, median (IQR)	50.0 (33.0–68.0)
Sex, n (%)
Female	926 (50.3)
Male	915 (49.7)
Comorbidities, n (%)
Hypertension	451 (24.5)
Diabetes mellitus	176 (9.6)
Coronary artery disease	130 (7.1)
Asthma or COPD	90 (4.9)
Stroke	73 (4.0)
Congestive heart failure	50 (2.7)
Malignancy	32 (1.7)
Chief complaint category, n (%)
Medical (non-surgical) problems	834 (45.3)
Neurologic problems	302 (16.4)
Cardiac problems	208 (11.3)
Trauma/orthopedic problems	196 (10.6)
Surgical (general surgery–type) problems	117 (6.4)
Pediatric problems	20 (1.1)
Obstetric/gynecologic problems	11 (0.6)
Other	153 (8.3)
Vital signs on admission, mean (SD)
Systolic blood pressure, mmHg	131.1 ± 21.1
Diastolic blood pressure, mmHg	76.4 ± 12.2
Heart rate, beats/min	88.6 ± 15.1
Respiratory rate, breaths/min	16.3 ± 3.5
Oxygen saturation, %	97.0 ± 3.2
Temperature, °C	36.6 ± 0.5
Glasgow Coma Scale, median (IQR)	15 (15–15)
Clinical outcomes, n (%)
Hospital admission (including ICU admission)	314 (17.1)
72-hour all-cause mortality	47 (2.6)

Note: Data are presented as number (n) and percentage (%), mean ± standard deviation (SD), and median with interquartile range (IQR). Comorbidities are not mutually exclusive

Abbreviations: COPD, chronic obstructive pulmonary disease; ICU, intensive care unit

Agreement in triage category assignment

Figure 1 shows agreement between the expert reference standard and GPT-5.5 Thinking triage category assignments. Overall agreement was substantial (quadratic-weighted Cohen κ = 0.721; 95% CI 0.691–0.740). The model achieved exact agreement (strict success) in 1141 cases (62.0%; 95% CI 59.7%–64.2%). Most disagreements were adjacent-category shifts, resulting in a within-1-level agreement of 99.0% (n = 1822; 95% CI 98.4%–99.3%). Misclassifications were asymmetric (Bowker test of symmetry, p < 0.001), with more overtriage than undertriage. Overtriage occurred in 650 cases (35.3%; 95% CI 33.2%–37.5%), and 97.7% of these were 1-level upgrades (n = 635; 95% CI 96.2%–98.6%). Undertriage occurred in 50 cases (2.7%; 95% CI 2.2%–3.4%). Dangerous undertriage occurred in 4 encounters (0.2%; 95% CI 0.1%–0.6%).

Fig. 1.

Agreement of the GPT-5.5 Thinking model with the expert reference standard for Manchester Triage System (MTS) category assignment. Note: The heat map shows the cross-tabulation of model-assigned MTS categories (columns) versus expert-assigned categories (rows) in the analytic cohort. Each cell reports the number of cases, n (%), with percentages calculated using the total number of evaluated cases (N = 1841). Darker shading indicates higher frequency. Diagonal cells indicate exact agreement; cells left of the diagonal indicate overtriage (model assigned a higher-urgency category), whereas cells right of the diagonal indicate undertriage (model assigned a lower-urgency category)

Table 2 demonstrates sensitivity analysis comparing safety-oriented and neutral prompting strategies. Strict success was slightly higher with the neutral prompt (63.9% vs. 62.0%, p < 0.001). Although the neutral prompting strategy was associated with a modest reduction in overtriage rates (32.6% vs. 35.3%, p < 0.001), it also resulted in higher undertriage rates (3.4% vs. 2.7%, p = 0.004). Dangerous undertriage remained rare under both prompting strategies but was numerically more frequent with the neutral prompt (0.4% vs. 0.2%).

Table 2.

Sensitivity analysis comparing safety-oriented and neutral prompting strategies

GPT-5.5 Thinking Assignment	with safety-oriented prompt	with neutral prompt
	n (%)	n (%)	p*
Assigned triage category
Red	67 (3.6)	62 (3.4)	0.180
Orange	361 (19.6)	332 (18.0)	< 0.001
Yellow	775 (42.1)	779 (42.3)	0.699
Green	638 (34.7)	668 (36.3)	< 0.001
Triage performance metrics
Strict success	1141 (62.0)	1177 (63.9)	< 0.001
Overtriage	650 (35.3)	601 (32.6)	< 0.001
Undertriage	50 (2.7)	63 (3.4)	0.004
Dangerous undertriage	4 (0.2)	7 (0.4)	0.180

Note: Data are presented as number (n) and percentage (%). *Comparisons between prompting strategies were performed using paired categorical tests (McNemar or Stuart–Maxwell tests, as appropriate)

Strict success was defined as an exact match between GPT-5.5 Thinking model-assigned and expert-assigned MTS categories (Red–Green)

Overtriage was defined as assignment to a higher-acuity category than the expert reference standard, whereas undertriage was defined as assignment to a lower-acuity category

Dangerous undertriage was defined as assignment two or more acuity levels lower than the expert reference standard

Subgroup analyses

Table 3 presents exact agreement (strict match) between the GPT-5.5 Thinking model and the expert reference standard across patient subgroups. Exact agreement did not differ by sex (p = 0.587) or across chief complaint categories (p = 0.188). However, exact agreement was significantly lower in patients aged ≥ 65 years compared with younger patients (57.1% vs. 64.2%, respectively; p = 0.013) (Table 3).

Table 3.

Exact agreement (strict match) between the GPT-5.5 Thinking model and the expert reference standard across patient subgroups

Category	Total cases n (%)	Strict success % (95% CI)	p-value*
Age distribution			0.013
< 65 years	1262 (68.6)	64.2% (61.5%–66.8%)
≥ 65 years	579 (31.4)	57.1% (52.9%–61.0%)
Sex			0.587
Male	915 (49.7)	62.2% (59.0%–65.3%)
Female	926 (50.3)	61.4% (58.3%–64.5%)
Chief complaint category			0.188
Medical (non-surgical) problems	834 (45.3)	57.4% (53.8%–60.3%)
Neurologic problems	302 (16.4)	63.9% (57.8%–68.6%)
Cardiac problems	208 (11.3)	60.1% (53.6%–65.9%)
Trauma/orthopedic problems	196 (10.6)	60.2% (53.4%–66.8%)
Surgical (general surgery–type) problems	117 (6.4)	70.9% (61.9%–78.4%)
Pediatric problems	20 (1.1)	65.0% (45.8%–83.4%)
Obstetric/gynecologic problems	11 (0.6)	45.5% (24.1%–73.3%)
Other	153 (8.3)	62.1% (54.3%–69.6%)

Note: Data are presented as number (n) and percentages with corresponding 95% confidence intervals (CIs). *Between-group comparisons were conducted using the χ² test or Fisher exact test for categorical variables, as appropriate

Strict success was defined as an exact match between GPT-5.5 Thinking model-assigned and expert-assigned MTS categories (Red–Green)

Prediction of admission and 72-hour mortality

Table 4 summarizes the performance of the GPT-5.5 Thinking model for predicting hospital admission and 72-hour mortality. For hospital admission, the model achieved an accuracy of 79.9%, with sensitivity of 83.8% and specificity of 79.1%. The PPV was 45.2% and the NPV was 95.9%. For 72-hour mortality, accuracy was 86.5%, with sensitivity of 72.3% and specificity of 86.9%. The PPV was 12.6% and the NPV was 99.1%.

Table 4.

Performance of the GPT-5.5 Thinking model for predicting hospital admission and 72-hour mortality

Metrics	Hospital admission prediction	72-hour mortality prediction
Accuracy	79.9% (78.0%–81.6%)	86.5% (84.9%–88.0%)
Sensitivity	83.8% (79.3%–87.5%)	72.3% (56.8%–81.8%)
Specificity	79.1% (76.9%–81.0%)	86.9% (85.3%–88.4%)
PPV	45.2% (41.2%–49.3%)	12.6% (9.2%–17.1%)
NPV	95.9% (94.7%–96.9%)	99.1% (98.5%–99.5%)

Note: Data are presented as percentages with corresponding 95% confidence intervals (CIs). For hospital admission, admission (including ICU admission) was treated as the positive outcome. For 72-hour mortality, death within 72 hours of ED presentation was treated as the positive outcome

Abbreviations: PPV, positive predictive value; NPV, negative predictive value

Observed outcomes by model-assigned triage category

Table 5 presents observed outcomes by GPT-5.5 Thinking assigned triage category. Among 1841 cases, the model assigned red category to 67 patients. In this group, 51 (76.1%) were admitted to hospital and 11 (16.4%) died within 72 h. Among 361 cases assigned Orange, 170 (47.1%) were admitted to hospital and 24 (6.6%) died within 72 h. In addition, among 775 cases assigned Yellow, 86 (11.1%) were admitted to hospital and 9 (1.1%) died within 72 h. Furthermore, Green category was assigned to 638 patients, with 7 (1.1%) hospital admissions and 3 (0.5%) 72-hour mortality.

Table 5.

Observed outcomes by GPT-5.5 Thinking-assigned Manchester Triage System category

GPT-5.5 Thinking Assignment	Cases n (%)	Hospital admission n (% within triage category)	72-hour mortality n (% within triage category)
Red	67 (3.6)	51 (76.1)	11 (16.4)
Orange	361 (19.6)	170 (47.1)	24 (6.6)
Yellow	775 (42.1)	86 (11.1)	9 (1.1)
Green	638 (34.7)	7 (1.1)	3 (0.5)

Note: Data are presented as number (n) and percentage (% within triage category). Hospital admission includes ICU admission

Discussion

In this selected analytic cohort, GPT-5.5 Thinking demonstrated substantial overall agreement with the expert reference standard for MTS-based ED triage. Exact category agreement was moderate, while most discordant classifications remained confined to adjacent triage levels. Misclassifications predominantly occurred in the direction of overtriage, whereas undertriage and dangerous undertriage were uncommon but not absent. These findings support consideration of the model only as a supervised decision-support tool within the population evaluated in this study. The model exhibited comparatively robust rule-out performance with respect to hospitalization and 72-hour mortality. However, its lower exact agreement in older adults suggests that subgroup-specific limitations should be carefully considered before real-world implementation.

In the emergency medicine literature, exact category agreement alone is no longer considered sufficient for evaluating AI-assisted triage systems. Current assessments also consider how large mistriage is, in which direction it occurs, and how it may affect patient safety [8, 9]. In our study, GPT-5.5 Thinking achieved exact agreement in 62.0% of cases. However, nearly all discordant classifications remained within adjacent triage levels, resulting in a within-1-level agreement of 99.0%. This pattern suggests that the model generally preserved the underlying hierarchy of urgency rather than assigning triage levels arbitrarily. This interpretation aligns with the findings of previous LLM-based triage studies. For instance, Masanneck et al. reported substantial agreement between GPT-4-based and professional MTS consensus, while also concluding that LLMs had not yet reached gold-standard triage performance [10]. Furthermore, Franc et al. employed CTAS-based simulated cases and discovered that the exact accuracy was lower, and there was considerable variability across prompts and repetitions. This finding indicates that exact agreement alone may not fully capture performance in ordinal triage systems [11]. Consequently, the model demonstrated a tendency to approximate the expert urgency ranking rather than significantly deviating from it.

In our cohort, overtriage occurred in 35.3% of cases, whereas undertriage occurred in 2.7%, and 97.7% of overtriage events were limited to a one-level upgrade. However, the predominance of overtriage in the primary analysis was partly attributable to the safety-oriented prompt and should not be interpreted solely as an inherent characteristic of the model. In the neutral-prompt sensitivity analysis, overtriage decreased to 32.6%, while undertriage increased to 3.4% and dangerous undertriage was numerically more frequent. Nevertheless, overtriage remained substantially more frequent than undertriage under both prompting conditions. Prior LLM-based triage studies have reported similar patterns. Masanneck et al. found that LLMs and GPT tended toward overtriage when compared with professional MTS raters [10]. Similarly, Franc et al. reported that GPT overtriaged more often than it undertriaged, with rates of 38.7% and 13.7%, respectively [11]. Zaboli et al. also observed that GPT showed a greater tendency to assign more urgent MTS codes than healthcare professionals [6]. Importantly, overtriage should not be regarded as operationally harmless. In real-world ED settings, repeated overtriage can lead to unnecessary prioritization of lower-acuity patients and place additional pressure on clinician time, monitoring capacity, and other limited ED resources [9]. Therefore, the high overtriage rate observed in the present study should be interpreted as a clinically relevant operational limitation, particularly in crowded ED settings where unnecessary escalation of acuity may increase clinician workload, monitoring demands, and competition for limited resources.

In our study, dangerous undertriage was very rare, though not non-existent. In the context of ED triage, the assignment of a high-risk patient to a lower-acuity category constitutes a form of mistriage that carries significant implications. This misclassification can result in delays in crucial clinical assessments, monitoring, and treatments. This concern is further substantiated by recent literature on AI-assisted triage [12]. Zaboli et al. found that, although GPT-4 showed a tendency toward overtriage, it may still produce undertriage in certain settings [6]. A recent systematic review and meta-analysis likewise emphasized that safety must be assessed alongside diagnostic accuracy because unsafe triage decisions remain a persistent concern in ED applications of GPT [4, 12]. Therefore, strong average agreement and favorable overall performance are not sufficient on their own. Even when rare, undertriage remains a critical constraint on autonomous clinical use and reinforces the need for close human oversight in any real-world implementation.

A salient finding of our study was the lower exact agreement observed in patients aged ≥ 65 years compared with younger adults. This finding may reflect the intrinsic complexity of geriatric ED triage. Older adults often present with atypical or nonspecific complaints and greater clinical complexity, making acuity assignment more context dependent. This challenge is well recognized in geriatric triage research. In a retrospective study of geriatric patients evaluated with a modified MTS, Li et al. noted that atypical presentation, multimorbidity, frailty, and cognitive impairment complicate accurate triage in older adults [13]. Similarly, Jang and Seo reported that undertriaged older patients frequently exhibited nonspecific neurological, cardiopulmonary, trauma-related, and respiratory complaints [14]. Beyond symptom atypia alone, age-related blunting of physiological responses and the effects of polypharmacy may further obscure illness severity at initial triage and increase the likelihood of false-low acuity assignment in some older patients. Evidence on LLM performance specifically within geriatric triage remains limited, as most published studies have focused on overall agreement, comparison with human raters, and general safety profiles rather than age-stratified performance [10, 11]. In this context, our findings suggest that LLM-based triage may be more vulnerable to error in older adults and that further age-specific validation is needed before these systems are considered for broader clinical use in geriatric populations.

For hospital admission and 72-hour mortality, NPV was higher than PPV. This suggests that the model was more effective in ruling out adverse short-term outcomes than in confidently ruling them in. The particularly high NPV for mortality should be interpreted cautiously because mortality prevalence was only 2.6%. These results may reflect the model’s ability to identify low-risk patients more than its ability to correctly identify true-positive cases. Williams et al. similarly noted that triage-based machine-learning models can show promising in silico performance but still require transparent development, temporal validation, and real-world testing before routine use [15]. Additionally, recent reviews have suggested that AI-based triage systems may support prioritization and resource planning, but concerns about generalizability and implementation remain unresolved [16]. While these tools may assist in identifying patients who are unlikely to require hospitalization or to experience early mortality, they should not substitute for the clinical judgment of healthcare professionals when making disposition decisions.

It should also be recognized that physiological risk stratification in emergency medicine is not unique to LLM-based triage systems. Established early warning tools have already demonstrated strong prognostic performance for short-term deterioration and mortality using simpler physiological variables and substantially lower implementation complexity [17]. Accordingly, AI-assisted MTS evaluation should currently be viewed as a complementary decision-support approach rather than a replacement for validated physiological early warning systems.

Limitations

Several limitations should be considered when interpreting the findings. First, this was a single-center study based on a consecutive 72-hour sample, which may limit generalizability to other institutions, seasonal case-mix patterns, weekday–weekend variations, or lower-volume ED settings. Second, patients classified as MTS Category 5 (Blue; nonurgent) by the expert reference standard were excluded from the analytic cohort. Because eligibility for the final analysis was partly determined by the reference-standard classification, this approach introduced selection and spectrum bias and restricted the evaluated acuity distribution to MTS Categories 1–4. The high number of excluded Category 5 presentations may also have affected the observed category distribution, agreement estimates, and predictive performance measures. Third, the primary performance estimates were obtained using a safety-oriented prompt that explicitly favored the higher-acuity category in borderline cases. This prompting choice was intended to reduce undertriage but also likely contributed to the observed predominance of overtriage. Furthermore, the relatively small number of mortality events may have limited the robustness of mortality prediction analyses and reduced the precision of outcome-related performance estimates. Fourth, the number of pediatric and obstetric patients in the analytic cohort was limited. Although the MTS contains dedicated flowcharts and discriminator-based considerations for these patient groups, the present study was not specifically designed to evaluate subgroup-specific triage performance in pediatric or obstetric populations. Therefore, caution is warranted when generalizing the findings to these clinical settings. Finally, the study population may have been influenced by prehospital selection mechanisms, including ambulance triage decisions, referral practices, and tertiary-care transfer patterns. These factors may have altered the acuity distribution of the analytic cohort and could limit generalizability to EDs with different prehospital systems or patient referral structures.

Conclusions

Within the selected analytic cohort, GPT-5.5 Thinking demonstrated substantial overall agreement with an expert reference standard for MTS-based ED triage and generally preserved the ordinal hierarchy of clinical urgency, with most disagreements occurring as adjacent-level shifts. However, the predominance of overtriage, the presence of rare but nonzero dangerous undertriage, and lower exact agreement in older adults indicate that LLM-based triage remains insufficient for autonomous clinical use. These findings support the potential role of GPT-5.5 Thinking as a supervised decision-support tool within human-in-the-loop ED triage workflows, but further prospective, multicenter, and subgroup-specific validation is required before real-world implementation.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

AI

Artificial intelligence

API

Application programming interface

CI

Confidence interval

COPD

Chronic obstructive pulmonary disease

CTAS

Canadian triage and acuity scale

ED

Emergency department

ESI

Emergency severity index

GPT

Generative pre-trained transformer

ICU

Intensive care unit

IQR

Interquartile range

IRB

Institutional review board

LLM

Large language model

MTS

Manchester triage system

NPV

Negative predictive value

PPV

Positive predictive value

SD

Standard deviation

STARD-AI

Standards for reporting diagnostic accuracy studies–artificial intelligence

TRIPOD-LLM

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis–large language models

Author contributions

S.T.F. and A.A. made the major contributions to the conception and design of the study, data interpretation, and drafting of the manuscript. O.S. contributed substantially to the study design, supervised the overall scientific process, critically revised the manuscript for important intellectual content, and provided senior academic oversight. T.A., M.F.O., S.F., M.S.H., E.S., M.O., E.B., N.K., T.S., A.F.C., Z.C., and Z.E. contributed to data collection, data curation, and review of the study materials. S.T.F. and A.A. wrote the main manuscript text. All authors reviewed the manuscript, approved the final version, and agree to be accountable for all aspects of the work.

Funding

The authors received no specific funding for this work.

Data availability

The datasets generated and/or analyzed during the current study are not publicly available because they are derived from hospital records and contain potentially sensitive patient-related information, but are available from the corresponding author on reasonable request and with permission of the relevant institutional authorities.

Declarations

Ethics approval and consent to participate

The study protocol was approved by the Institutional Review Board of Haseki Training and Research Hospital, Istanbul, Türkiye (Decision No: 53-2026). The study was conducted in accordance with the ethical principles of the Declaration of Helsinki (2024 revision). Because the data and accompanying text did not contain potentially identifiable patient information, the Institutional Review Board waived the requirement for informed consent for the review of medical records.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.