Comparing the text-based diagnostic reasoning performance of emergency medicine physicians and large language models in both definitive and differential diagnoses using standardized clinical vignettes: a preliminary study

Abstract

Background

Timely and highly accurate diagnoses by physicians play a crucial role in improving the quality and effectiveness of patient treatment outcomes. Currently, the use of artificial intelligence capabilities in this area has garnered the attention of many health science researchers. Therefore, the main goal of this preliminary study was to compare the text-based diagnostic reasoning performance of emergency medicine physicians and large language models in definitive and differential diagnoses using standardized clinical vignettes.

Methods

This descriptive comparative study evaluated the diagnostic accuracy of 10 emergency medicine physicians and 4 large language models (LLMs)—ChatGPT (GPT-5.2), Gemini 3, Microsoft Copilot (GPT-4), and Claude Opus 4.1—using 10 standardized clinical vignettes. All LLMs were accessed via official web interfaces. Clinical vignettes were developed from emergency department presentations, validated by an expert panel, and presented as text-only inputs to all evaluators. Diagnostic accuracy was assessed using a standardized scoring protocol: definitive diagnoses required an exact match with expert-derived reference standards, while differential diagnoses required ≥ 3 matches. Overall diagnostic accuracy was the primary outcome. Data were analyzed using Pearson’s Chi-square test, McNemar’s test, and Generalized Estimating Equation (GEE) logistic regression with Bonferroni correction (SPSS version 28).

Results

A total of 280 diagnostic evaluations (10 clinical cases assessed by 14 evaluators across 2 diagnosis types) were analyzed. Overall diagnostic accuracy was 61.79%. AI models demonstrated significantly higher overall accuracy (73.75%) compared to emergency medicine physicians (57.00%, p = 0.014). Across all evaluators, definitive diagnoses were more accurate than differential diagnoses (70.71% vs. 52.86%). Generalized Estimating Equation (GEE) analysis revealed a significant interaction between evaluator group and diagnosis type (p = 0.036). Specifically, physicians experienced a significant decline in accuracy when providing differential diagnoses compared to definitive diagnoses (45.0% vs. 69.0%, p = 0.003; remained significant after Bonferroni correction). In contrast, AI models maintained consistently high accuracy across both diagnosis types, with no significant difference between definitive (75.0%) and differential (72.5%) diagnoses (p = 1.000).

Conclusion

Large language models outperformed emergency medicine physicians in overall diagnostic accuracy and demonstrated superior consistency across different types of diagnostic tasks. While human physicians struggled significantly with differential diagnoses, AI models maintained high and stable performance regardless of the diagnostic complexity. These findings indicate that AI possesses robust pattern-recognition and reasoning capabilities, suggesting it could serve as a highly reliable clinical decision support tool, particularly in complex scenarios requiring differential diagnostic reasoning. Due to study limitations, such as the small number of clinical scenarios and assessors, these findings should be interpreted with significant caution.

Introduction

Artificial intelligence (AI) has become a fundamental part of modern medicine. This massive technological transformation, leveraging machine learning, deep neural networks, and big data analytics, is a critical step toward transforming clinical practice, improving patient outcomes, and optimizing healthcare systems [1, 2]. AI can drive precision medicine by integrating genomic, clinical, and real-world data, and provide tailored treatments by predicting the course of disease [3, 4]. AI-powered tools, particularly those that employ deep learning, are highly effective at analyzing large, complex datasets, including medical images (X-rays, CT scans), vital signs, and laboratory results. These systems can rapidly detect a range of diseases, often achieving accuracy rates of 85–90% in imaging-based diagnoses [5, 6].

Emergency medicine is a critical field where AI can significantly enhance diagnostic accuracy and accelerate response times [7]. In emergencies, every second can impact a patient’s outcome. Specialists, including physicians, often have to work with inconsistent and incomplete data while managing demanding schedules. This creates a pressing need for tools that facilitate rapid and accurate decision-making. In this context, large language models (LLMs) such as ChatGPT-4, Gemini, Claude, and Copilot have emerged as valuable diagnostic aids, providing new opportunities to improve the efficiency of diagnostic systems [8]. These AI-driven systems can deliver timely and accurate predictions, enhance standardization in clinical assessments, and help identify subtle imaging patterns or connectivity changes that might be overlooked by human practitioners [9].

AI-based clinical decision support (CDS) systems are increasingly integrated into emergency medicine to assist with high-stakes, time-sensitive tasks such as diagnosis, triage, and risk stratification [7, 10]. These systems leverage machine learning to synthesize vast volumes of clinical data into actionable insights, aiming to improve diagnostic accuracy and optimize workflow in the chaotic environment of the emergency department (ED) [10, 11].

Some empirical evidence on the application of AI in emergency medicine suggests that AI-enabled automated diagnostic support facilitates early treatment. AI also achieves high accuracy in medical imaging and condition recognition, supporting physicians in challenging cases [12]. AI systems can process complex datasets rapidly, supporting high-stakes decisions in time-sensitive settings [13]. In this regard, studies have been conducted on the recognition accuracy of LLMs. Recent systematic reviews show that physicians generally achieve higher overall diagnostic accuracy than LLMs, although LLMs outperform clinicians in approximately one-third of evaluated scenarios and, in some complex clinical vignettes, frontier models such as Claude-3.5 Sonnet and GPT-4o perform comparably to or better than physicians, with top-three diagnostic accuracies approaching 100% [14–16]. In complex critical illness, advanced LLMs have also demonstrated substantially higher diagnostic accuracy than unaided residents and can improve resident performance to 58% while nearly halving diagnostic time when used as decision support [17]. However, evidence indicates that simply providing physicians with access to an LLM does not necessarily improve diagnostic reasoning, highlighting that effective clinical benefit depends on careful workflow integration, appropriate oversight, and minimizing automation bias rather than relying on the models alone [18, 19].

In the emergency department, physicians typically assess patients with limited information and must make sensitive decisions based on probabilities and risk assessments. Therefore, the hospital emergency department can be a unique environment for leveraging AI capabilities, as the technology has great potential for clinical predictions during triage and for analyzing a variety of patient factors simultaneously [20, 21].

Emergency medicine represents a uniquely appropriate domain for evaluating AI diagnostic performance due to its inherent characteristics: undifferentiated patient presentations, time pressure, incomplete information, and the need for rapid decision-making under uncertainty [11, 22]. Diagnostic errors in the emergency department are a known patient safety concern, often stemming from cognitive biases, knowledge gaps, or system-level constraints [23]. AI systems may mitigate some of these limitations by providing instant access to extensive medical knowledge and generating comprehensive differential diagnoses [10, 22]. Consequently, assessing AI performance in this high-stakes setting offers valuable insights into both the potential and limitations of current generative AI technologies in frontline clinical care [7, 11].

Despite ongoing research into the applications of AI in medical science, concerns persist regarding its use and implications for patient safety, as the scientific evidence supporting its widespread implementation remains limited. Also, the interdisciplinary nature of the scientific field of AI, lack of training & AI literacy, accuracy, bias, and safety concerns, organizational/cultural resistance, and the lack of reliable and accessible sources of peer-reviewed studies have made it difficult for medical practitioners to have a clear understanding of AI interventions [24, 25]. Evidence shows that LLMs such as ChatGPT, while reliable for standard emergency protocols and capable of accurate imaging analysis, also have weaknesses. In ambiguous or highly complex cases, AI may lack nuances and context-sensitive reasoning, highlighting the need for ongoing clinician oversight and multidisciplinary input [26].

Despite promising advances, a critical knowledge gap remains regarding the comparative diagnostic performance of state-of-the-art LLMs versus experienced emergency medicine physicians. While several studies have begun to evaluate LLMs for medical reasoning and triage [16], few directly compare their diagnostic accuracy against that of board-certified emergency physicians across a representative spectrum of undifferentiated ED presentations using standardized methodologies [27].

Given the importance of conducting further studies on the role of AI with an emphasis on LLMs in various fields of medical sciences, the present study was designed and implemented. Therefore, this study aims to compare the accuracy of definitive and differential diagnoses made by emergency medicine specialists and LLMs using standardized clinical vignettes. With this aim in mind, we seek to answer the following question: Is there a difference in the accuracy of definitive versus differential diagnosis between emergency medicine specialists and large language models regarding standardized clinical vignettes?

Methods

Study design and setting

This descriptive comparative study was conducted between June and November 2025. The study sample comprised 14 evaluators: 10 human emergency medicine specialists and 4 AI models. The study flow diagram is presented in Fig. 1.

Fig. 1.

Study flow diagram

The selected sample size is sufficient and methodologically sound for the study’s goals. Because the four AI models represent the entire population of major available LLMs, traditional power calculations are unnecessary. For the human cohort, the 10 physicians align with similar exploratory studies [8, 28] and provide adequate power to detect meaningful diagnostic differences, driven by the large effect sizes typically seen in AI-versus-human comparisons.

Evaluator selection

Human evaluators were recruited using convenience sampling from emergency medicine specialists at Besat Hospital in Tehran, Iran. Inclusion criteria required: (1) board certification in emergency medicine; (2) ≥ 2 years of clinical experience in the emergency department (ED); (3) direct involvement in acute diagnostic workups, including the formulation of definitive and differential diagnoses; and (4) provision of informed consent. Four commercially available LLMs were selected as AI evaluators: ChatGPT (GPT-5.2, OpenAI), Gemini 3 (Google), Microsoft Copilot (GPT-4, Microsoft), and Claude Opus 4.1 (Anthropic). Models were chosen based on their clinical prevalence, high accessibility, and established accuracy, consistency, transparency, and security per current literature [29]. Continuous availability throughout the analysis period was also required. Although Microsoft Copilot utilizes a GPT-4 foundation, it was evaluated as a distinct platform due to its unique system prompts, integrated retrieval-augmented generation (web search), and proprietary safety filters, which significantly alter clinical outputs compared to raw API implementations. Specific AI configurations and parameters were logged to ensure reproducibility (Table 1).

Table 1.

The specific parameters for each model

Parameter	ChatGPT (OpenAI)	Google Gemini 3	Microsoft Copilot	Claude Opus 4.1
Underlying Model Name	GPT-5.2	Gemini 3	GPT-4	Claude Opus 4.1
Model Version	(Exact version string)	(Exact version string)	(Exact version string)	(Exact version string)
Access Date	(July 20–30, 2025)	(July 20–30, 2025)	(July 20–30, 2025)	(July 20–30, 2025)
Interface Used	Web Interface /	Web Interface	Web Interface	Web Interface
Reasoning Mode	Standard / Enabled	Standard / Enabled	Standard / Enabled	Standard / Enabled
Multimodal Capability	Text-only / Text+Image	Text-only / Text+Image	Text-only / Text+Image	Text-only / Text+Image
Temperature / Parameters	Default / 0.0	Default / 0.0	Default / 0.0	Default / 0.0

Clinical vignettes: development and validation

Ten clinical vignettes were developed using purposive sampling from patients presenting to the emergency department in 2025 (Table 2). These cases represent curated examples selected for their diagnostic complexity rather than a consecutive series of emergency department patient encounters. Cases were intentionally selected to reflect high-acuity, diagnostically complex presentations commonly encountered in emergency practice (e.g., acute abdominal pain, chest pain, dyspnea). Clinical data were systematically structured using the OLD CHARTS mnemonic (Onset, Location, Duration, Character, Aggravating factors, Relieving factors, Timing, Severity). Straightforward conditions typically diagnosed via immediate objective testing (e.g., uncomplicated urinary tract infections, simple cellulitis, isolated electrolyte disorders) were excluded, as they do not adequately assess advanced clinical reasoning or differential diagnosis generation. To ensure the scenarios appropriately represented emergency medicine practice, multiple validation procedures were implemented: The expert panel assessed whether each scenario appeared to be a realistic and authentic representation of actual emergency department patient presentations. All scenarios were confirmed to have high face validity, meaning they were judged by the expert panel to be plausible, clinically coherent, and representative of real-world emergency medicine cases.

Table 2.

Results of definitive and differential diagnosis of designed scenarios based on common symptoms of hospital emergency department visitors

Patient	Patient Problem	Definitive Diagnosis	Differential Diagnosis
57-year-old man, job: staffer	My chest feels heavy, and it’s hard to breathe, especially when I move. I’ve been sweating a lot, and I feel weak.	Pulmonary embolism	Acute coronary syndrome, aortic dissection, pleurisy, pneumothorax, esophageal spasm
60-year-old man, retired employee	Suddenly, I felt a sharp pain in the middle of my chest, and my back was shooting. I felt dizzy and felt like I had low blood pressure.	Aortic dissection	Acute coronary syndrome, pulmonary embolism, pneumothorax, esophageal spasm, esophageal rupture
30-year-old male, athlete	I constantly have severe nausea and diarrhea, I can’t eat anything, and I feel extremely physically weak.	Viral gastroenteritis	Food poisoning, infectious colitis, pancreatitis, intestinal obstruction, irritable bowel syndrome
45-year-old woman, teacher	For several months now, I’ve been experiencing a burning sensation in the middle of my stomach after eating, especially when I eat fatty foods or when I’m stressed. I feel heavy.	Gastroesophageal reflux disease (GERD)	Peptic ulcer, gastritis, cholecystitis, pancreatitis, esophageal spasm
65-year-old man, retired employee	I haven’t been able to move my arms or legs since this morning, and one side of my body seems to be weak. It’s also becoming difficult for me to speak.	Ischemic stroke	Intracerebral hemorrhage, transient ischemic attack (TIA), vertebrobasilar insufficiency, multiple sclerosis, myasthenia gravis
48-year-old man, engineer	When I woke up this morning, I had a severe headache. My vision was blurry, and the light was very bothersome.	Migraine	Subarachnoid hemorrhage, increased intracranial pressure, optic neuritis, sinusitis, and hypertension
40-year-old man, employee	I have severe pain in my side, and I feel the pain all the way down to my groin. I don’t feel any improvement.	Renal Colic	Kidney stones, urinary tract infection, ureteral torsion, appendicitis, diverticulitis
18-year-old boy, athlete	I don’t feel well at all. I feel weak, I’m sweating coldly, I’m pale, and I’m dizzy.	Hypovolemic shock	Sepsis, cardiogenic shock, anaphylaxis, internal bleeding, pulmonary thromboembolism
40-year-old man, driver	After the accident, I feel a lot of pain in my stomach, and I feel dizzy and weak.	Spleen rupture	Internal bleeding, liver rupture, hemothorax, kidney damage, leg fracture
70-year-old woman, housewife	I have been coughing constantly for two days, feeling short of breath, and physically weak.	Bacterial pneumonia	Heart failure, chronic obstructive pulmonary disease (COPD), tuberculosis, acute bronchitis, respiratory distress syndrome

A panel of three board-certified emergency physicians (each with ≥ 5 years of clinical experience) independently reviewed all scenarios to confirm that they comprehensively represented high-acuity, diagnostically complex presentations across major clinical domains of emergency medicine. The panel verified that the selected cases reflected the spectrum of diagnostic complexity typically encountered by emergency medicine specialists. Before the main study, the scenarios were pilot-tested with two emergency medicine physicians who were not included in the final study sample. The pilot test confirmed that the scenarios were clearly worded, contained sufficient clinical information for diagnostic decision-making, and were appropriately challenging for board-certified emergency medicine specialists. Minor refinements to scenario wording were made based on pilot feedback to improve clarity.

Following pilot testing, the expert panel conducted a final review to establish the definitive and differential diagnoses for each case, serving as the reference “ground truth.” The panel confirmed clinical accuracy and resolved discrepancies through structured consensus. The consensus process involved all three experts reviewing the scenarios together, discussing their individual assessments, and reaching unanimous agreement on the reference diagnoses. Cases that lacked consensus among the expert panel were excluded from the final set of ten scenarios. Reference diagnoses were further corroborated using Rosen’s Emergency Medicine and Churchill’s Differential Diagnosis to ensure alignment with established emergency medicine literature. Case difficulty was stratified based on Emergency Severity Index (ESI) triage levels, comorbidity burden, and expert panel ratings on a 5-point Likert scale (1 = very easy, 5 = very complex), confirming a moderate-to-high diagnostic complexity across all ten scenarios.

Prompting strategy and AI interaction protocol

All clinical scenarios were provided to the AI models as text-only inputs. Multimodal inputs (e.g., ECGs, radiographs, laboratory result images) were intentionally excluded to isolate and evaluate the models’ pure textual clinical reasoning capabilities based on history, physical examination findings, and laboratory values, without the confounding variable of image-recognition accuracy. All clinical information was manually transcribed from patient medical records into standardized text vignettes by the research team before presentation to the AI models. A standardized, zero-shot prompt was used to minimize prompt-induced bias, and all models were run at their default temperatures to reflect real-world usage. The exact prompt was: “As an experienced emergency medicine specialist with a thorough understanding of scientific emergency medicine textbooks, please provide five differential diagnoses and one definitive diagnosis for the following clinical scenario.”

Data collection and scoring rubric

Diagnostic responses were collected sequentially from human physicians and then from the AI models. A predefined, binary scoring protocol was applied to all outputs. For definitive diagnoses, accuracy required an exact match with the reference standard. For differential diagnoses, a response was scored correct if ≥ 3 of the provided conditions matched the expert-derived reference list. To ensure consistency in evaluating diagnostic responses, a standardized scoring protocol was implemented. Before scoring, the expert panel developed a terminology mapping document that identified accepted synonyms, abbreviations, and equivalent diagnostic terms. This document was used by both independent reviewers to ensure uniform interpretation of diagnostic terminology.

Responses were scored as follows: Exact matches to the reference diagnosis were scored as correct. Clinically equivalent terms (as defined in the terminology mapping document) were scored as correct. Partial matches or terminology variations that were less specific than the reference diagnosis were reviewed by both independent reviewers and scored as correct only if both reviewers agreed the response was clinically acceptable in the context of the case. Overly broad or nonspecific terms were scored as incorrect.

To handle ambiguous AI outputs, a standardized rubric was applied: (1) If multiple diagnoses were listed, correctness required the target diagnosis to be ranked first; (2) If the AI’s conclusion did not map to the reference standard, it was scored incorrect; (3) If explanatory text conflicted with the final stated diagnosis, the conclusive statement took precedence; (4) If only a differential list was provided without a primary diagnosis, correctness required the target diagnosis to appear within the top three.

Total diagnostic accuracy was calculated by summing the scores across the ten clinical scenarios. All responses were independently evaluated by two physicians with expertise in the relevant specialty, who were blinded to whether the responses originated from physicians or AI models. Inter-rater reliability was calculated using Cohen’s kappa, and discrepancies were resolved through consensus or adjudication by a third senior emergency physician.

Representative case illustration

To enhance methodological transparency and illustrate the diagnostic evaluation process, a representative clinical scenario with comparative outputs from both physician and AI evaluators is presented in Table 3. This example demonstrates the application of the standardized prompting strategy and the variability in diagnostic reasoning between human and AI evaluators under identical clinical information.

Table 3.

Representative clinical vignette with Comparative Diagnostic Outputs

Component	Content
Clinical Vignette	A 45-year-old female teacher presents with a several-month history of burning epigastric pain, exacerbated by fatty foods and psychological stress, accompanied by postprandial heaviness. No alarming symptoms reported.
Reference Standard	Definitive Diagnosis: Gastroesophageal reflux disease (GERD) Differential Diagnoses: Peptic ulcer disease, gastritis, cholecystitis, pancreatitis, esophageal spasm
AI Model Output	Definitive Diagnosis: Functional dyspepsia Differential Diagnoses: Peptic ulcer disease, GERD, chronic pancreatitis, cholecystitis (4 provided instead of 5 requested)
Physician Output	Definitive Diagnosis: Chronic pancreatitis Differential Diagnoses: Cholecystitis, gastroenteritis, irritable bowel syndrome, intestinal obstruction, peptic ulcer disease
Scoring Outcome	AI Model: Definitive diagnosis incorrect; differential diagnosis correct (3/5 matched: PUD, GERD, cholecystitis) Physician: Definitive diagnosis incorrect; differential diagnosis incorrect (only 1/5 matched: cholecystitis)

Data gathering tools

Human Evaluator Data Collection: Diagnostic responses from emergency medicine specialists were collected using a standardized diagnosis registration form specifically designed for this study. The form was structured to capture comprehensive clinical information and consisted of the following sections:

Patient Demographics: Age, and gender. Presenting Complaint: Primary reason for emergency department visit. Clinical History: Systematically organized using the OLD CHARTS mnemonic. Diagnostic Assessment: Two dedicated columns requiring the evaluator to provide: One definitive diagnosis and five differential diagnoses (ranked in order of clinical suspicion).

Before implementation, the registration form underwent content validation by the expert panel of three board-certified emergency physicians to ensure clarity, completeness, and alignment with standard diagnostic documentation practices. A pilot test was conducted with two physicians not included in the final sample to identify any ambiguities, and minor refinements were made based on their feedback.

AI Model Data Collection: Responses from the four LLMs were collected using a standardized computational setup to ensure consistency and reproducibility. Data collection was performed on a dedicated workstation with the following specifications: Intel Core i7 processor, 16 GB RAM, Windows 11 operating system. All AI interactions were conducted via the official web-based chat interfaces of each model (ChatGPT at chat.openai.com, Gemini at gemini.google.com, Microsoft Copilot at copilot.microsoft.com, and Claude at claude.ai), rather than API implementations. This approach was deliberately chosen to evaluate the models as they are typically accessed by end-users in clinical practice, including their default system prompts, proprietary safety filters, and platform-specific configurations. All models were accessed using their default temperature settings and standard user accounts without specialized developer configurations. To ensure methodological rigor, an AI specialist with expertise in prompt engineering was consulted during the study design phase to optimize prompt structure and minimize potential biases. The same standardized diagnosis registration form used for human evaluators was adapted to systematically record AI model outputs, ensuring an identical data structure across all 14 evaluators.

Data Management and Quality Control: All responses—both human and AI—were recorded contemporaneously and entered into a secure, password-protected database. Each evaluator’s responses were assigned a unique identifier to maintain consistency during scoring. To ensure data integrity, all entries were double-checked by two independent research assistants, and any discrepancies were resolved through consensus.

Outcomes

The primary outcome was overall diagnostic accuracy, defined as the proportion of correct responses (both definitive and differential diagnoses combined) out of the total number of evaluations performed by each evaluator group (AI models vs. emergency medicine physicians).

Secondary outcomes included. Definitive diagnosis accuracy: The proportion of cases in which the evaluator’s definitive diagnosis exactly matched the reference standard. Differential diagnosis accuracy: The proportion of cases in which the evaluator’s differential diagnosis included ≥ 3 conditions matching the expert-derived reference list. Diagnostic performance by diagnosis type: Comparison of accuracy between definitive and differential diagnoses within each evaluator group. Interaction effects: Assessment of whether the difference in performance between AI models and physicians varied by diagnosis type (definitive vs. differential).

Data analysis

Data were analyzed using IBM SPSS Statistics version 28. Categorical variables, including diagnostic accuracy, were summarized as frequencies and percentages. Diagnostic accuracy was calculated as the proportion of correct responses. For statistical modeling, the dependent variable was defined as binary diagnostic accuracy, with correct diagnoses coded as 1 and incorrect diagnoses as 0. To compare the overall diagnostic accuracy between the two independent groups (AI models vs. physicians), the Pearson Chi-square test was initially employed as an unadjusted analysis. Because each evaluator assessed the same set of clinical cases, the observations for definitive and differential diagnoses were paired. Therefore, McNemar’s test was utilized to perform paired comparisons of accuracy between definitive and differential diagnoses within each group (physicians and AI models separately). Given the hierarchical and clustered nature of the study design—where multiple evaluations were nested within each evaluator (10 cases per evaluator) and each case was evaluated by multiple raters—the assumption of independence required for standard tests was violated. To appropriately account for these within-subject correlations and provide robust estimates, a Generalized Estimating Equation (GEE) logistic regression model was employed as the primary inferential analysis. The GEE model is specifically designed for analyzing repeated measures or clustered binary outcome data. The model was constructed with the binary diagnostic score (correct vs. incorrect) as the dependent variable, and evaluator group (AI vs. physician), diagnosis type (definitive vs. differential), and the interaction between group and diagnosis type as independent predictors. An exchangeable working correlation structure was assumed to account for the repeated measures. All statistical tests were two-sided, and a p-value of < 0.05 was considered statistically significant.

Ethical requirements

This study was conducted in accordance with the principles of the Declaration of Helsinki and was approved by the Ethics Committee of the Army University of Medical Sciences (AUMS), Tehran, Iran (IR.AJAUMS.REC.1403.053). Written informed consent was obtained from all participating emergency medicine physicians before their involvement in the study. The study’s objectives and the methods for data collection, processing, and storage were explained. Participation was voluntary. Participants were assured that the study results would be used solely for scientific research purposes. To ensure strict patient privacy and confidentiality, all clinical vignettes were rigorously de-identified before use. Any protected health information (PHI), including patient names, exact dates, national IDs, or other unique identifiers, was removed or altered. Furthermore, to comply with ethical guidelines for artificial intelligence research, no real, identifiable patient data was directly input into public AI interfaces; only sanitized, text-based clinical vignettes were used for evaluation.

Results

A total of 280 diagnostic evaluations were analyzed, comprising assessments of 10 clinical cases by 14 evaluators (10 emergency medicine physicians and 4 AI models). The mean age of the physicians was 42.00 ± 10.91 years. Each evaluator provided both a definitive and a differential diagnosis for every case. Overall, the combined diagnostic accuracy across all evaluators and cases was 61.79% (173/280). The overall diagnostic accuracy was higher for AI models (73.75%; 59/80) than for physicians (57.00%; 114/200). When stratified by diagnosis type, definitive diagnoses were generally more accurate than differential diagnoses across the entire cohort (70.71% vs. 52.86%, respectively) (Table 4).

Table 4.

Descriptive statistics—overall diagnostic accuracy and by type of diagnosis

Group	Correct	Total	Accuracy (%)
Physicians	114	200	57.00%
AI Models	59	80	73.75%
Overall	173	280	61.79%
Diagnosis Type	Correct	Total	Accuracy (%)
Definitive	99	140	70.71%
Differential	74	140	52.86%

The results of the Pearson Chi-Square test indicated that the overall difference between the two groups was statistically significant (χ² = 6.10, p = 0.014) (Table 5). Cross-tabulation revealed that while both groups performed better on definitive diagnoses, the accuracy gap between definitive and differential diagnoses was much more pronounced among physicians (69.0% vs. 45.0%) than among AI models (75.0% vs. 72.5%) (Table 6).

Table 5.

Chi-square test—comparing AI vs. physicians (overall accuracy)

Statistic	Value
Pearson Chi-Square (χ²)	6.10
p-value	0.014

Table 6.

Cross-tabulation—accuracy by group × diagnosis type

Group	Diagnosis	Correct	Total	Accuracy (%)
Physicians	Definitive	69	100	69.0%
Physicians	Differential	45	100	45.0%
AI Models	Definitive	30	40	75.0%
AI Models	Differential	29	40	72.5%

Post-hoc paired comparisons using McNemar’s test were conducted to evaluate this interaction. To control for multiple comparisons, a Bonferroni correction was applied, setting the significance threshold at α = 0.025. The analysis demonstrated that physicians experienced a significant drop in accuracy when providing differential diagnoses compared to definitive diagnoses (45.0% vs. 69.0%, p = 0.003, which remained significant after Bonferroni correction). In contrast, AI models maintained a consistently high accuracy across both diagnosis types, with no significant difference between definitive (75.0%) and differential (72.5%) diagnoses (p = 1.000) (Table 7).

Table 7.

McNemar’s test—definitive vs. differential diagnosis (paired comparison)

Group	McNemar’s χ²	p-value
Physicians	8.82	0.003
AI Models	0.00	1.000

Because each evaluator assessed multiple clinical cases, the observations are inherently correlated. To account for this repeated-measures design and provide a robust analysis, a Generalized Estimating Equation (GEE) logistic regression model was employed. The GEE analysis confirmed a significant main effect for diagnosis type, indicating that differential diagnoses were significantly more difficult to get correct than definitive diagnoses across all evaluators (β = −1.001, p < 0.001) (Table 8).

Table 8.

GEE logistic regression (accounting for repeated measures)

Predictor	Coefficient (β)	Std. Error	z-value	p-value	95% CI	Interpretation
Intercept	0.800	0.123	6.52	< 0.001	[0.559, 1.041]	Baseline log-odds
Group (AI vs. Doctor)	0.299	0.181	1.65	0.100	[-0.057, 0.654]	Main effect of group
Diagnosis Type (Differential vs. Definitive)	−1.001	0.249	−4.02	< 0.001	[-1.489, − 0.512]	Differential diagnoses are harder overall
Group × Diagnosis Type (Interaction)	0.872	0.416	2.09	0.036	[0.056, 1.688]	Significant interaction

Discussion

The present study aimed to compare the text-based diagnostic reasoning performance of emergency medicine physicians and LLMs in both definitive and differential diagnoses using standardized clinical vignettes. To this end, the performance of ten physicians and four LLMs was compared across ten clinical vignettes based on real-life cases. The findings of this exploratory study suggest that LLMs may achieve higher overall diagnostic accuracy than emergency medicine physicians in structured clinical vignettes. More notably, LLMs exhibited consistent performance across both definitive and differential diagnostic tasks, whereas physicians demonstrated a significant decline in accuracy when formulating differential diagnoses. However, these findings must be interpreted with considerable caution given the study’s limited sample size, curated case selection, and exploratory design.

Our results are broadly consistent with a growing body of evidence indicating that frontier LLMs can match or exceed physician performance in structured diagnostic tasks. Recent systematic reviews and meta-analyses have reported that generative AI models demonstrate diagnostic performance comparable to, and in certain contexts superior to, that of physicians [14–16]. Sorich et al. specifically noted that the latest-generation models have substantially narrowed the performance gap with physician evaluators in both triage and diagnostic accuracy tasks [16].

However, the literature presents a more nuanced picture when the physician’s expertise level is considered. Evidence suggests that while expert physicians may outperform AI by a statistically significant margin in certain contexts, non-expert clinicians and LLMs perform similarly, with pooled diagnostic accuracy rates that do not differ significantly between groups [14, 15]. This pattern has been observed across various clinical domains, including hepatocellular carcinoma diagnosis [30] and inflammatory rheumatic diseases [31], reinforcing the context-dependent nature of these comparisons. Importantly, in clinical case-based evaluations, physicians have generally performed better overall, although certain generative AI models have demonstrated comparable diagnostic accuracy [32].

The observed lack of a significant performance decline in LLMs between definitive and differential diagnoses may suggest that their underlying knowledge retrieval and pattern-recognition mechanisms are relatively stable across different diagnostic tasks. Unlike human clinicians, AI models do not appear susceptible to cognitive fatigue, premature closure, or anchoring bias when generating and evaluating multiple alternative hypotheses. This preliminary observation aligns with the perspective that LLMs may process complex clinical information and navigate differential reasoning with a consistency that contrasts with known vulnerabilities in human cognitive processing.

Conversely, the significant decline in physicians’ accuracy when providing differential diagnoses may reflect the inherent challenges of human clinical reasoning under specific conditions. In the high-stakes, fast-paced environment of emergency medicine, the cognitive load required to systematically articulate and validate multiple differential diagnoses may strain working memory, potentially contributing to decreased accuracy.

Despite promising findings regarding diagnostic accuracy, the integration of AI into clinical practice faces substantial challenges that warrant careful consideration. Patient and clinician trust in AI for medical diagnoses remains generally lower than trust in human physicians, particularly for high-risk conditions [33]. Evidence suggests that while patients may expect comparable performance from both AI and physicians, they initially trust human diagnoses more and prefer them for second opinions in serious cases. Notably, trust in AI increases significantly when patients are actively involved in selecting AI-assisted diagnoses, indicating that perceived control over medical decisions plays a crucial role in acceptance [33]. Simply demonstrating that AI may outperform physicians in certain tasks does not automatically translate into increased trust without meaningful patient involvement in the decision-making process.

AI systems in healthcare also face significant practical and ethical challenges, including dependence on high-quality, representative datasets; risks of algorithmic bias; potential for over-reliance that may diminish clinician expertise; lack of emotional interaction between patients and providers; reduced effectiveness for atypical presentations or nuanced clinical judgment requiring intuition; cybersecurity vulnerabilities; and high implementation costs [34, 35]. These limitations underscore the need for cautious, evidence-based integration of AI tools into clinical workflows.

Furthermore, patient-centered research indicates that trust in AI diagnoses is partly limited by concerns about explanation and transparency in diagnostic decision-making [30]. While providing explanations of AI diagnostic reasoning can enhance patients’ functional, relational, and emotional assessments of the consultation, complete replacement of human physicians by AI remains unlikely given the vital emotional and relational roles that human clinicians fulfill [32].

The prevailing perspective in the literature, consistent with our findings, is that AI functions most appropriately as a supportive tool rather than a replacement for clinicians. The relational, contextual, and empathetic dimensions of human diagnosis cannot currently be replicated by AI, although AI may enhance accuracy, efficiency, and outcomes when used as an adjunct in diagnosis and treatment planning. The role of AI is thus synergistic—amplifying medical capabilities without entirely replacing physician roles [31, 33].

Physicians may benefit from using AI outputs as decision support rather than as substitutes for clinical judgment. Expert physicians are advised to cross-check patient data, carefully review medical guidelines and established standards, and integrate AI-generated insights with their clinical expertise to arrive at accurate diagnoses and treatment decisions. This evolving interaction has the potential to improve the diagnostic process while preserving the care and oversight provided by human physicians [32].

This study provides a foundational comparison of diagnostic accuracy between emergency physicians and AI models. Future research should address the current gaps through several key avenues. First, studies must utilize larger, multicenter datasets encompassing a wide, unselected variety of clinical presentations to mitigate selection bias and improve generalizability. Second, future evaluations should transition from curated clinical vignettes to real-world, unstructured emergency department cases to better reflect the ecological validity and messy reality of everyday clinical practice. Third, prospective validation studies are needed to assess AI performance in real-time clinical workflows and measure its actual impact on patient outcomes. Finally, the development and implementation of standardized AI benchmarking frameworks are essential to ensure consistent, reproducible, and independent evaluations of emerging medical AI tools across different institutions.

This study has several limitations that must be considered when interpreting the findings. A primary limitation of this study is the relatively small sample size, consisting of 10 clinical cases evaluated by 10 emergency medicine physicians and 4 AI models. Because the cases were specifically selected for this study, they may not fully represent the complexity, acuity, or variety of patients seen in routine practice, thereby limiting the generalizability of our results. Furthermore, this constrained sample size precluded robust stratified subgroup analyses (e.g., by physician experience or disease category), as dividing the data would result in severely underpowered comparisons with highly unstable estimates. Therefore, the results should be interpreted as exploratory and hypothesis-generating rather than definitive.

While our primary analytical approach utilizing Generalized Estimating Equations (GEE) is robust for the overall dataset, future multi-center studies with a substantially larger volume of clinical cases and a broader panel of evaluators are warranted. Such studies would provide the necessary statistical power to adequately explore these stratified analyses and further elucidate the nuances of diagnostic accuracy across different clinical and demographic subgroups. There are inherent limitations related to the evaluation of artificial intelligence. Although four distinct AI models were assessed, there is potential overlap in their underlying foundational architectures and training data, which may limit the true independence of the AI evaluators. Additionally, the diagnostic performance of LLMs is highly dependent on prompt wording and engineering; variations in how clinical information is structured and presented could significantly alter the models’ outputs.

Another limitation of this study was that both physician and AI evaluators were restricted to text-only clinical vignettes, without access to physical examination, ECG, laboratory results, imaging, or iterative patient interaction. These necessarily disadvantaged physicians, who in real-world practice integrate multiple data sources to arrive at diagnoses. Therefore, this study compared text-based diagnostic reasoning under standardized conditions rather than real-world clinical diagnosis. This design was necessary to isolate pure reasoning capabilities and ensure standardized comparison, but it may underestimate physicians’ actual diagnostic performance. Future studies should evaluate performance using real-world emergency department cases with complete multimodal data.

Finally, AI models are rapidly evolving and continuously updated. The performance captured in this study reflects a specific snapshot in time and may not accurately reflect the current or future capabilities of these systems as they undergo further training and optimization. Future studies with larger cohorts of physicians, a broader range of clinical scenarios, and inclusion of emerging AI models are needed to confirm these findings and to perform adequately powered confirmatory analyses.

Conclusion

In this exploratory study, LLMs demonstrated significantly higher overall diagnostic accuracy compared to emergency medicine physicians. Notably, while physicians experienced a significant decline in accuracy when formulating differential diagnoses, AI models maintained consistently high performance across both definitive and differential diagnostic tasks. These findings highlight the potential of AI to serve as a highly valuable complementary tool in the emergency department. By assisting in the rapid and accurate generation of differential diagnoses, AI can help mitigate cognitive biases and broaden clinical consideration. AI is a powerful tool that enhances physicians’ capabilities in diagnosis by improving accuracy, efficiency, and early detection, but clinical validation and physician oversight are essential for safe and effective use in practice. It is imperative to emphasize that AI must function strictly as a clinical decision support system rather than an autonomous diagnostician. Final diagnostic and treatment decisions must unequivocally be made by human physicians, who integrate AI-generated insights with comprehensive clinical evaluation, physical examination, and nuanced patient context. Nevertheless, as an exploratory study with a limited number of clinical scenarios, the current results provide only preliminary observations. Future research must focus on confirming these findings through larger, multi-center, prospectively designed studies utilizing real-world emergency department cases and standardized AI evaluation and benchmarking methodologies. Significant risks and challenges regarding errors, accountability, transparency, and ethical use must be carefully addressed to mitigate threats. Collaboration between clinicians, developers, and policymakers is crucial to ensure equitable, safe, and trustworthy AI applications.

Abbreviations

AI

Artificial Intelligence

LLMs

Large Language Models

OLD CHART

Onset, Location, Duration, Character, aggravating factors, relieving factors, Timing, and Severity

AUMS

Army University of Medical Sciences

Author contributions

MASH, HJ, AO, and SR design and conceptualize. MASH, HJ, AO, and SMA Methodology, analyze, and interoperate. HJ, MSH, SR, RV, and MSH Data collection. MASH, HJ, AO, SMA, and MSH wrote the original draft. All authors read and approved the final submitted version of the manuscript.

Funding

Not applicable.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

The study received approval from the National Committee for Ethics in Research at Army University of Medical Sciences (AUMS) (IR.AJAUMS.REC.1403.053) and was conducted in full accordance with the Declaration of Helsinki. All participants gave informed consent to take part. No additional permissions were necessary.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Data avilability

The datasets used during this study are available from the corresponding author on reasonable request.