Can AI match emergency physicians in managing common emergency cases? A comparative performance evaluation

Abstract

Background

Large language models (LLMs) such as ChatGPT are increasingly explored for clinical decision support. However, their performance in high-stakes emergency scenarios remains underexamined. This study aimed to evaluate ChatGPT’s diagnostic and therapeutic accuracy compared to a board-certified emergency physician across diverse emergency cases.

Methods

This comparative study was conducted using 15 standardized emergency scenarios sourced from validated academic platforms (Geeky Medics, Life in the Fast Lane, Emergency Medicine Cases). ChatGPT (GPT-4) and a physician independently evaluated each case based on five predefined parameters: diagnosis, investigations, initial treatment, clinical safety, and decision-making complexity. Cases were scored out of 5. Concordance was categorized as high (5/5), moderate (4/5), or low (≤ 3/5). Wilson confidence intervals (95%) were calculated for each concordance category.

Results

ChatGPT achieved high concordance (5/5) in 8 cases (53.3%, 95% CI: 27.6–77.0%), moderate concordance (4/5) in 4 cases (26.7%, CI: 10.3–55.4%), and low concordance (≤ 3/5) in 3 cases (20.0%, CI: 6.0–45.6%). Performance was strongest in structured, protocol-based conditions such as STEMI, DKA, and asthma. Lower performance was observed in complex scenarios like stroke, trauma with shock, and mixed acid-base disturbances.

Conclusion

ChatGPT showed strong alignment with emergency physician decisions in structured scenarios but lacked reliability in complex cases. While AI may enhance decision-making and education, it cannot replace the clinical reasoning of human physicians. Its role is best framed as a supportive tool rather than a substitute.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12873-025-01303-y.

Introduction

Artificial intelligence (AI) is increasingly influencing medical decision-making across various disciplines, particularly in high-pressure settings like emergency departments (EDs), where clinical efficiency and diagnostic accuracy are crucial [1]. With the recent surge in AI integration into clinical workflows, large language models (LLMs) like ChatGPT have emerged as novel tools capable of analyzing medical data, generating diagnostic hypotheses, and suggesting evidence-based treatments [2].

In recent years, LLMs such as ChatGPT have been increasingly applied across various medical areas, including clinical decision-making, test interpretation, documentation, and diagnostic reasoning [3]. In emergency medicine specifically, their use has expanded to include triage support, clinical documentation, patient follow-up, and image interpretation tasks [4, 5]. These developments show a growing interest in integrating AI tools into routine workflows to enhance clinical efficiency and support diagnostic accuracy.

In parallel with these developments, the growing integration of AI into medical practice has also raised questions and concerns about whether such technologies might replace human physicians [6]. This topic has been frequently discussed in both medical and public settings in recent years. However, recent perspectives emphasize that AI should be considered a complementary tool rather than a replacement for clinical expertise, especially in high-acuity and rapidly evolving environments such as emergency departments [7]. Recent research has found that many clinical specialists acknowledge the potential of AI but strongly believe that it cannot fully replace the diagnostic reasoning and decision-making capabilities of experienced physicians [8, 9].

Despite the growing integration of AI tools, current evidence highlights significant limitations and challenges when LLMs are applied in complex medical decision-making. These models often lack access to patient-specific clinical details at the point of care, struggle with interpreting nuanced physical examination findings, or fail to maintain consistency across highly variable clinical scenarios. Moreover, their tendency to generate hallucinated or outdated information remains a substantial concern [10, 11]. It’s crucial to distinguish these capabilities and experimental settings from actual clinical practice, especially in emergency departments, where physicians make rapid decisions based on incomplete or evolving information, interpret non-verbal cues, and rely heavily on situational awareness and bedside intuition [12, 13]. These human elements are inherently difficult to replicate in static, text-based case vignettes. Such limitations underscore the need for cautious implementation and emphasize the importance of evaluating AI models within realistic, domain-specific contexts before considering broader clinical use [14].

This study compares the diagnostic reasoning and treatment recommendations of ChatGPT with those of a board-certified emergency physician in terms of the diagnostic accuracy of common emergency cases.

Materials and methods

This study compared the diagnostic performance of ChatGPT and an emergency physician using 15 structured clinical scenarios. These cases were selected to reflect common emergency presentations frequently encountered in the emergency department. The goal was to assess how effectively ChatGPT could interpret and manage these scenarios using the clinical information provided.

Case selection and preparation

Fifteen common emergency cases were meticulously selected from academically validated, open-access resources, including Geeky Medics, Life in the Fast Lane, and Emergency Medicine Cases [15–17]. Each case was anonymized and presented as a typical emergency department scenario to GPT.

In the selection process, specific attention was paid to ensuring diversity in three key areas: clinical complexity (e.g., straightforward vs. diagnostically challenging cases), urgency (e.g., time-critical vs. non-urgent conditions), and specialty domain (e.g., cardiology, neurology, toxicology, and gastroenterology). This diversity aimed to enhance the generalizability and real-world relevance of the findings.

The final set of cases encompassed a wide spectrum of emergency scenarios, including both protocol-based conditions (e.g., ST-elevation myocardial infarction, diabetic ketoacidosis, asthma, anaphylaxis, acute kidney injury, and gastrointestinal bleeding) and complex, multifactorial situations requiring advanced clinical reasoning (e.g., stroke, sepsis, trauma with shock, pulmonary embolism, mixed acid-base disturbances, opioid overdose, and hypertensive emergency).

Each case was carefully prepared in a structured format reflecting typical emergency clinical presentations. These case summaries included essential elements such as patient history, presenting symptoms, and key examination findings. For consistency and standardization, all cases were presented to ChatGPT by inputting the full scenario as a single prompt in plain text. No prior conversation history or external information was provided to the model, ensuring unbiased response generation.

Study design

Each case was assessed by a board-certified emergency physician with experience in managing emergency conditions. For each case, ChatGPT’s response (OpenAI; GPT-4, March 2025 version) and the expected clinical approach were compared across five clear criteria that reflect key parts of decision-making in emergency care:

• Likely diagnosis: Was the main and differential diagnosis correct and complete?

• Recommended investigations: Were the tests appropriate and listed in the right order?

• Treatment plan: Was the first treatment step safe, correct, and in line with common practice?

• Clinical safety and applicability: Could these suggestions be safely used in a real emergency department?

• Complexity of decision-making: Did the answer show understanding of the case’s challenges and need for flexible thinking?

Each part was scored as 1 if ChatGPT gave a complete and correct answer, and 0 if the answer was wrong, missing, or not detailed enough.

A total score out of 5 was given for each case. All cases were reviewed separately, and there was no outside influence during scoring.

For clarity, the overall results were grouped as:

• High concordance: 5 out of 5.

• Moderate concordance: 4 out of 5.

• Low concordance: 3 or fewer out of 5.

The full text of each case scenario, along with the specific prompts provided to ChatGPT, its responses, and detailed scoring rubrics with explanations for each evaluation parameter, are all provided in Appendix A.

ChatGPT evaluation and prompting procedure

Each clinical case was manually entered into ChatGPT (OpenAI; GPT-4) as a single, isolated prompt. No prior conversation history or contextual information was provided, ensuring that each case was evaluated independently.

The prompt used in each case was:

What is the most likely diagnosis?

Which tests would you request?

What is the initial treatment plan?

For example:

(e.g., A 28-year-old male presents to the emergency department approximately 15 min after eating. He complains of sudden-onset generalized itching, hives (widespread red, itchy welts on his skin), a feeling of throat tightness, difficulty breathing, and dizziness. On examination, his vital signs are BP 80/50 mmHg, HR 120 bpm, RR 30 breaths/min, and SpO₂ 90% on room air. He has diffuse urticaria, angioedema of the lips and eyes, and audible wheezing on chest auscultation. He appears anxious and is speaking in a hoarse voice. What is the most likely diagnosis? Which tests would you request? What is the initial treatment plan?).

ChatGPT’s responses were evaluated by the emergency physician using the five predefined assessment parameters described in the Study Design section.

Emergency physician evaluation

The evaluations were conducted by a single board-certified emergency medicine specialist with extensive clinical experience in managing common emergency conditions. Emergency physicians are trained to follow standardized, evidence-based approaches to assess and manage such cases. The same specialist reviewed all AI and human responses in order to maintain consistency across the assessments.

Statistical analysis

The proportion of concordance for each case type was calculated based on the total number of cases achieving high (5/5), moderate (4/5), or low (≤ 3/5) concordance scores. To assess the precision of these proportions and account for the binomial nature of the data, 95% Wilson confidence intervals (CIs) were calculated. A lower bound below 60% was used as an additional indicator to flag concordance categories that may reflect substantially low diagnostic agreement, highlighting clinically significant limitations in performance.

Ethical approval

This study exclusively involved theoretical cases with no real patient data, thus ethical approval was not required. All cases were structured using anonymized data from academic sources.

Results

Across the 15 structured emergency scenarios evaluated, ChatGPT demonstrated varying levels of concordance with the emergency physician, reflecting the nature and complexity of each case. High concordance (5/5) was observed in 8 cases (53.3%), moderate concordance (4/5) in 4 cases (26.7%), and low concordance (≤ 3/5) in 3 cases (20%). These findings indicate that while ChatGPT’s performance was strongly aligned with human clinical judgment in algorithmic and protocol-driven scenarios, it was less reliable in cases demanding nuanced reasoning or dynamic assessment. Detailed diagnostic and treatment concordance is summarized in Table 1, and Wilson confidence intervals are provided in Table 2.

Table 1.

Diagnostic concordance between ChatGPT and emergency physician by case type

Case Type	Concordance Level	Comments
STEMI	High (5/5)	Accurate diagnosis, prompt ECG and treatment
DKA	High (5/5)	Recognized insulin and fluid therapy needs
Asthma	High (5/5)	Identified bronchodilator use and oxygen therapy
Panic Attack	High (5/5)	Distinguished from life-threatening conditions
Anaphylaxis	High (5/5)	Correct epinephrine and airway approach
Gastroenteritis + Dehydration	High (5/5)	Focused on rehydration and symptom relief
AKI	High (5/5)	Correct diagnosis, timely intervention and management
Acute GI Bleeding	High (5/5)	Correct diagnosis, prompt intervention and stabilization
Pulmonary Embolism	Moderate (4/5)	Missed Wells criteria; otherwise appropriate
Sepsis	Moderate (4/5)	Early antibiotics suggested; less accurate severity
Opioid Overdose	Moderate (4/5)	Correct naloxone use; insufficient airway detail
Hypertensive Emergency	Moderate (4/5)	Treatment safe but lacked end-organ assessment
Stroke	Low (2/5)	Missed neurologic exam detail and time sensitivity
Trauma with Shock	Low (2/5)	Underprioritized ABCs and hemorrhage control
Mixed Acid-Base Disorders	Low (2/5)	Provided general ideas without accurate calculation

Concordance reflects agreement in diagnostic and management decisions

Table 2.

ChatGPT’s diagnostic performance by concordance category with 95% Wilson confidence intervals

Concordance Category	Number of Cases (n = 15)	Proportion (%)	95% Wilson CI
High (5/5)	8	53.3%	27.6–77.0%
Moderate (4/5)	4	26.7%	10.3–55.4%
Low (≤ 3/5)	3	20.0%	6.0–45.6%

The table presents the distribution of chatgpt’s diagnostic performance across high, moderate, and low concordance levels, along with corresponding 95% Wilson confidence intervals

The eight cases demonstrating high diagnostic concordance (5/5) between ChatGPT and the emergency physician typically involved structured, guideline-driven conditions, where established protocols were readily applicable. For example, in the ST-elevation myocardial infarction (STEMI) case, ChatGPT accurately identified the infarct location, recommended a right-sided ECG, and prioritized early PCI and dual antiplatelet therapy—mirroring real-world best practices. Similarly, in the diabetic ketoacidosis (DKA) scenario, the AI proposed a complete diagnostic workup, fluid resuscitation, insulin therapy, and electrolyte monitoring consistent with international guidelines.

In other acute presentations like asthma exacerbation, anaphylaxis, acute gastrointestinal bleeding, and acute kidney injury (AKI), ChatGPT reliably suggested appropriate initial investigations, bronchodilator or epinephrine use, fluid management strategies, and ongoing monitoring. Importantly, clinical safety and decision-making complexity were consistently rated as adequate, with no missed red flags or unsafe therapeutic steps. Even in less structured cases such as panic attack or gastroenteritis with dehydration, the AI correctly ruled out life-threatening conditions and provided safe, patient-centered management plans. These results suggest that in predictable, algorithm-based emergencies, ChatGPT can provide clinically appropriate recommendations comparable to physician judgment.

Among the four cases with moderate concordance (4/5), ChatGPT generally provided appropriate diagnoses and treatment suggestions but demonstrated limitations in clinical nuance or structured risk assessment. In the pulmonary embolism case, the model correctly identified the diagnosis and recommended relevant imaging and anticoagulation. However, it did not apply a validated clinical decision rule such as the Wells score, which is essential for risk stratification before proceeding with advanced imaging in stable patients. Similarly, in the sepsis case, although early antibiotics and fluid resuscitation were recommended, the response lacked structured severity assessment using tools like SOFA or qSOFA, and failed to consider cranial imaging for altered mental status, reflecting a partial gap in holistic evaluation.

In the hypertensive emergency scenario, ChatGPT correctly recognized the condition and recommended cautious blood pressure lowering. Yet, the absence of key assessments such as fundoscopy or detailed end-organ damage evaluation reduced the clinical depth of the response. Finally, in the opioid overdose case, while naloxone administration and diagnostic workup were correctly suggested, the AI failed to explicitly address airway protection despite the patient’s shallow breathing and decreased consciousness, which presents a potential safety concern in real-life emergency settings. These findings highlight that in moderately complex cases, ChatGPT may generate broadly acceptable responses, but may omit critical steps that require deeper clinical insight or structured prioritization.

In contrast, three scenarios demonstrated low concordance (≤ 3/5), highlighting key clinical and analytical limitations in ChatGPT’s responses. In the acute ischemic stroke case, although the model correctly identified the diagnosis and recommended appropriate investigations (CT, MRI, coagulation studies), it inappropriately suggested intravenous thrombolysis (tPA) despite an unknown time of symptom onset, representing a significant deviation from clinical guidelines. Furthermore, the response lacked nuanced reasoning regarding wake-up strokes, failing to incorporate eligibility criteria or advanced imaging interpretation for decision-making.

Similarly, in the trauma with hemorrhagic shock case, the model recognized the likely intra-abdominal bleeding and proposed basic investigations, but did not prioritize blood transfusion in the initial treatment plan, despite overt signs of hypovolemia. Critically, airway protection strategies were omitted, and the overall approach lacked urgency—both of which are essential in trauma resuscitation.

In the mixed acid-base disturbance scenario following cardiac arrest, the AI correctly identified the presence of metabolic acidosis but failed to characterize the mixed nature of the disorder. It did not perform or mention key acid-base calculations (e.g., anion gap, delta ratio, Winter’s formula), which are vital for guiding appropriate treatment. This superficial approach also impacted clinical safety and decision-making complexity ratings.

Overall, ChatGPT achieved high concordance (5/5) in 53.3% of the cases (8 out of 15), correctly addressing all five evaluation parameters in over half of the structured emergency scenarios. Moderate agreement (4/5) was observed in 26.7% of the cases (4 out of 15), where responses were generally sound but lacked clinical nuance or omitted key considerations. Low concordance (≤ 3/5) occurred in 20.0% of the cases (3 out of 15), reflecting significant limitations in reasoning and safety in more complex presentations. These findings, supported by Wilson confidence intervals (high: 95% CI: 27.6–77.0%; moderate: 10.3–55.4%; low: 6.0–45.6%), indicate that while ChatGPT performs reliably in structured, algorithm-based emergencies, its effectiveness declines in high-risk, ambiguous scenarios—underscoring the need for critical human oversight in real-world emergency care.

Discussion

This study supports the growing body of evidence that large language models (LLMs) such as ChatGPT can offer substantial diagnostic and therapeutic guidance in emergency medicine—particularly for structured, guideline-driven scenarios [2]. High concordance in STEMI, DKA, and asthma suggests that AI tools can successfully mimic the decision-making of trained physicians in cases governed by clinical algorithms. This is further supported by recent research (Hoppe et al.2024) showing that ChatGPT-4 outperformed emergency physicians in diagnostic accuracy across a retrospective case series, highlighting its strength in algorithm-based decision-making [12].

One of the major weaknesses of AI remains the inability to perform physical examinations, interpret non-verbal cues, or adapt to dynamic patient deterioration. For example, neurologic localization in stroke or prioritization of airway management in trauma require real-time, bedside judgment. Moreover, the time constraints of emergency care often make AI consultation impractical during critical, high-acuity events [10, 18]. In cases with lower concordance, the limitations of ChatGPT’s recommendations could translate into clinically significant consequences in real-world emergency care. For instance, the model’s suggestion of intravenous thrombolysis in a stroke patient with an unknown time of onset deviated from current guidelines and could have increased the risk of intracerebral hemorrhage. Similarly, failure to prioritize airway management in trauma and opioid overdose scenarios—where patients presented with signs of respiratory compromise—could result in hypoxia, aspiration, or cardiac arrest if applied without human oversight. In the mixed acid-base disturbance scenario, ChatGPT failed to perform key calculations such as the anion gap or delta ratio, which are essential for identifying the underlying etiology and tailoring appropriate treatment. This observation is further supported by the findings of Turan et al. (2025), who evaluated ChatGPT-4’s performance in interpreting arterial blood gas (ABG) parameters. Their study demonstrated high accuracy for basic markers such as pH, oxygenation, and compensation status. However, it also highlighted potential risks of overtreatment, including inappropriate recommendations for bicarbonate therapy. In our case-based evaluation, ChatGPT achieved only 2/5 in the mixed acid-base disturbance scenario, underscoring its limitations in analyzing multifactorial and context-dependent emergencies. These differences may reflect the model’s difficulty with real-time reasoning in clinically complex situations [19]. These examples highlight the potential dangers of applying AI-generated responses without critical evaluation, especially in high-acuity or time-sensitive situations. Recent evidence, such as that by Bushuven et al. (2023), supports these concerns, noting that use of large language models in emergency care without appropriate safeguards may lead to unsafe decisions or missed priorities [20].Furthermore, De Simone et al. (2025) emphasize that while AI has transformative potential, its application in acute and dynamic settings like emergency surgery still raises unresolved ethical and safety concerns that must be addressed before routine adoption [20, 21].

A significant concern with large language models (LLMs), particularly in high-stakes environments like emergency medicine, is the phenomenon of hallucinations—instances where the AI generates confident yet incorrect or fabricated information not supported by training data or clinical context. While our study did not explicitly categorize responses as hallucinations, several critical errors—such as recommending thrombolysis in a stroke case with unknown onset or failing to perform essential acid-base calculations like the anion gap or delta ratio—may reflect hallucination-like behavior or underlying limitations in reasoning capacity [22, 23]. These issues were particularly evident in low-concordance scenarios such as stroke, trauma with shock, and mixed acid-base disturbances. Such confident but erroneous outputs can be dangerous in acute care settings, where rapid and accurate decision-making is vital. Addressing the risk of hallucinations and ensuring model reliability remain essential challenges for the safe integration of LLMs into clinical practice [24].

Recent literature evaluating the clinical performance of GPT-based models reflects similar patterns to those observed in our study. Several investigations have highlighted that large language models tend to perform well in structured, algorithm-driven cases. For instance, Jo et al. (2024) reported that GPT-4’s medical recommendations were often comparable to those of human experts, particularly in clearly defined scenarios [25]. Likewise, Hirosawa et al. (2024) demonstrated ChatGPT-4’s diagnostic accuracy in experimental settings, especially when reasoning pathways were standardized and protocol-based [26].

Conversely, these studies also emphasize a consistent decline in AI performance under ambiguous or high-stakes conditions requiring nuanced clinical judgment. Öncü et al. (2025) found that while ChatGPT-4o supported intern physicians during case simulations, it occasionally failed to account for contextual complexity [27]. Similarly, Hao et al. (2025) observed that ChatGPT offered useful support in musculoskeletal decision-making, but struggled in atypical presentations [28].

Our findings align with this pattern. ChatGPT achieved high diagnostic concordance in 53.3% of emergency cases, but moderate or low agreement in nearly half of the scenarios—particularly in cases involving stroke, trauma, or mixed acid-base disturbances. As shown in our Wilson analysis, this variability underscores a key limitation: while ChatGPT can reliably follow clinical algorithms, it remains insufficient when deeper interpretive reasoning or prioritization under uncertainty is required. These results reinforce the view that large language models, though promising, should serve as adjuncts rather than autonomous decision-makers in emergency care.

Nonetheless, AI may offer substantial benefits in non-critical scenarios, in junior physician education, and in resource-constrained settings. It can reinforce evidence-based practices and support decision-making when appropriately supervised. However, as our findings and recent studies indicate, AI is not yet poised to fully replace human specialists in complex, high-stakes clinical scenarios, which highlights the continuing necessity of critical human oversight [8, 29].

Limitations

This study has several limitations. Firstly, using theoretical scenarios instead of real-time patient interactions limits the ecological validity of our findings. We couldn’t assess AI’s response to dynamic changes, incomplete information, or nuanced clinical cues crucial in actual emergencies.

Secondly, the evaluation process relied on a single board-certified emergency physician to score all responses. While this approach ensured consistency in criteria application, it means our findings are primarily based on a single expert’s judgment. The absence of inter-rater reliability assessment implies that the generalizability and reproducibility of our results are inherently restricted to this individual evaluation. Future studies incorporating multiple independent reviewers would enhance the objectivity and robustness of such assessments.

Lastly, practical aspects of AI use, such as user interface design, response times, and integration into real-world clinical workflows, were not assessed in this study. Future research should explore these critical operational considerations for a full understanding of AI’s utility and safety in emergency care.

Future directions

While this study utilized standardized theoretical cases, future research should prioritize the application of AI tools in real-world clinical settings. Prospective studies involving live patient interactions, the integration of AI into electronic health records (EHR), and performance evaluations under real emergency conditions are essential to validate these findings and further assess the practical utility of AI in emergency care settings. In addition, incorporating multiple independent reviewers in future evaluations may enhance methodological rigor and improve the reliability of performance assessments.

Conclusion

ChatGPT demonstrated strong diagnostic and therapeutic alignment with an experienced emergency physician in structured cases. While it is not a substitute for clinical expertise, AI provides valuable support in decision-making and education. Its role is best defined as a complementary tool, particularly in settings with limited resources or for training purposes. Despite its potential, AI cannot replace the clinical reasoning and judgment of emergency physicians, especially in complex or unpredictable real-life scenarios. Future studies should assess AI tools using real-time patient data to better understand their utility and limitations in live emergency care settings.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

AI

Artificial Intelligence

ED

Emergency Department

LLM

Large Language Model

STEMI

ST-Elevation Myocardial Infarction

DKA

Diabetic Ketoacidosis

ABG

Arterial Blood Gas

CT

Computed Tomography

MRI

Magnetic Resonance Imaging

SOFA

Sequential Organ Failure Assessment

qSOFA

Quick Sequential Organ Failure Assessment

EHR

Electronic Health Record

PCI

Percutaneous Coronary Intervention

CI

Confidence Interval

GCS

Glasgow Coma Scale

BP

Blood Pressure

HR

Heart Rate

RR

Respiratory Rate

SpO₂

Peripheral Capillary Oxygen Saturation

AKI

Acute Kidney Injury

tPA

Tissue Plasminogen Activator

ECG

Electrocardiogram

ABCs

Airway, Breathing, Circulation

Author contributions

M.G. conceived the study, designed the methodology, performed the data analysis, wrote the manuscript, and reviewed and approved the final manuscript.

Funding

The author received no financial support for the research, authorship, or publication of this article.

Data availability

All case scenarios used in this study, along with the specific prompts provided to ChatGPT, its responses, and the detailed scoring rubrics, are included in Appendix A. All materials were derived from publicly available, open-access educational sources. No real patient data were used.

Declarations

Ethics approval and consent to participate

Not applicable. No human participants were involved in the study, and all data were obtained from open-access educational platforms.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.