Large language models versus human examinee performance on Israeli anesthesiology board examinations

Abstract

Large Language Models (LLMs) demonstrate increasing capabilities in medical knowledge assessment, yet limitations remain in cross-population validation, direct human-AI comparisons, and evaluation of newer models in anesthesiology contexts. This study addresses these gaps by conducting a head-to-head comparison between newer LLMs and human examinees on official Israeli multiple-choice board examinations. We evaluated two LLMs (Claude 3.7 Sonnet and ChatGPT-4) against anonymized aggregate data from 381 examinees on three consecutive official Israeli anesthesiology board examinations (2023–2024), comprising 450 multiple-choice questions stratified by difficulty, discrimination ability, and topic. Each model was tested twice per exam. Claude 3.7 Sonnet achieved 73.67% accuracy, significantly outperforming both human examinees (62.77%, P < 0.001) and ChatGPT-4 (64.44%, P < 0.001). However, both LLMs performed below the upper quartile of human performance (78.05%). While LLMs excelled on easy questions and theoretical domains like cardiac physiology (Claude: 96.88%, ChatGPT-4: 81.25%), they showed lower performance in areas such as ambulatory (Claude: 30.00%, ChatGPT-4: 10.00%) and regional anesthesia (Claude: 44.44%, ChatGPT-4: 38.89%). Human examinees demonstrated consistent performance across all domains, whereas LLMs showed extreme variability. Self-consistency was substantial for both LLMs (κ = 0.66–0.68), but agreement with human responses was moderate (κ = 0.34–0.39). While advanced LLMs currently exceed average examinee performance on anesthesiology board examinations, they fall short of top-quartile examinees at present and demonstrate significant performance variability across different topic areas.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-45411-6.

Introduction

The emergence of sophisticated Large Language Models (LLMs) has generated significant interest regarding their potential applications in medicine¹. Their ability to process and interpret complex information has prompted investigations into their utility for medical education and assessment^2–4.

Comparing LLM performance against expert practitioners offers valuable insights into artificial intelligence’s (AI) current capabilities within medical practice⁵. Research has increasingly focused on evaluating these models through medical licensing exams⁶, and various specialty board exams^7–12, using these assessments as surrogate markers for specialized medical knowledge.

Previous research has examined LLM performance on anesthesiology written board exams, including the American Board of Anesthesiology (ABA)^13–15 and the Fellowship of the Royal College of Anaesthetists (FRCA) exams^16,17, demonstrating varying performance levels (approximately 45% to 80% accuracy), with notable differences between earlier and more advanced models.

These studies, while valuable, have several limitations. First, they have primarily focused on ABA and FRCA examinations, limiting cross-population validation in other healthcare systems. Second, most studies relied on practice materials rather than official exam questions. Third, no studies have directly compared LLM performance with actual human examinee results. Last, they consistently demonstrate variable performance across different LLMs, and the rapid evolution of LLM technology necessitates ongoing evaluation of newer models that remain untested in anesthesiology contexts. This study addresses these limitations by conducting a head-to-head comparison between newer LLMs and human examinees on official Israeli board examinations.

Methods

Design and ethics

This study compared LLM performance with human examinee results on official Israeli anesthesiology board examinations. The Israeli Anesthesiology Examination Committee of the Israel Medical Association Scientific Council provided official exam materials, answer keys, and anonymized aggregate examinee data, and granted permission for research and publication. This study was reviewed by the Rabin Medical Center Institutional Review Board (Prof. Ran Tur-Kaspa, Chairman) and the Hadassah Medical Center Institutional Review Board (Prof. Dina Ben Yehuda, Chairman), which waived the requirement for ethical approval and informed consent as the study involved only anonymized group-level statistics with no identifiable human subjects. All methods were carried out in accordance with relevant guidelines and regulations.

Study materials and data collection

The Israeli anesthesiology residency is a five-year program, with board certification requiring two sequential examinations. The Step 1 examination is a written MCQ examination administered to residents who have completed at least 2 years of their residency program. Each Step 1 exam consisted of 150 MCQs with four answer options. All questions are developed by a national committee of senior anesthesiologists using standardized guidelines, and the exam is administered in Hebrew. Questions are categorized by topic, with the majority based on chapters from Miller’s Anesthesia, 9th Edition textbook¹⁸, which served as the official reference designated by the Israeli Anesthesiology Examination Committee for these examination sessions, supplemented by additional categories for advanced cardiac life support (ACLS) and general clinical knowledge. The passing threshold for the Step 1 examination is 65%. The Step 2 examination is an oral board examination, consisting of simulation and structured oral stations covering various anesthesiology subspecialties, available to residents who have passed Step 1 and completed at least 4 years of residency.

The present study evaluated only the Step 1 MCQ examination. The analysis included three consecutive Step 1 exam sessions, conducted in 2023 and twice in 2024. The examination encompasses both foundational/scientific and clinical management content. The content distribution was consistent across all three sessions, with questions spanning basic sciences including physiology and pharmacology (33.3%), clinical foundations including monitoring, airway management, and regional anesthesia (30.7%), subspecialty anesthesia including cardiac, obstetric, trauma, pediatric, and critical care (28.9%), and ACLS and general clinical knowledge (7.1%).

Two LLMs were evaluated: ChatGPT-4 (OpenAI^®) and Claude 3.7 Sonnet (Anthropic^®). Both models were accessed through their respective paid subscription web-based interfaces (ChatGPT Plus and Claude Pro) using default settings and parameters. Both LLMs were selected as representative of advanced language models with strong multilingual capabilities, including Hebrew language processing. All LLM queries were conducted in April 2025. Each model was tested twice on each exam to assess both accuracy and self-consistency. Questions were presented to the LLMs one at a time, with each individual question submitted in a separate, newly initiated chat session to prevent memory retention between questions, using a single standardized user prompt: ‘Please select the most appropriate answer,’ with no system prompt or additional instructions provided. For a small number of linked questions (3–4 per exam) where a subsequent item depended on the clinical scenario but not on the correct answer to the preceding question, both questions were submitted within the same chat session to preserve clinical context. All questions were presented in Hebrew and administered exactly as they appeared in the original examinations, in the same order as the original test. In terms of question format, each item was either text only or included visual elements such as medical images (chest X-rays, CT scans, echocardiograms), physiological traces (ECG strips, capnography waveforms, arterial line traces), diagnostic charts (pressure-volume curves, TEG traces), and anatomical diagrams. Both LLMs were evaluated using their native multimodal capabilities when visual content was present. Following the completion of the first full exam run, all chat history was deleted before beginning the second attempt to prevent any memory retention from the first attempt. The memory feature, which allows models to retain information across separate chat sessions, was explicitly disabled in the settings of both platforms throughout the study. This procedure was applied across all three exams, resulting in a total of six runs per model (two per exam).

The collected dataset included, for each question: topic classification, the model’s first and second responses, correctness based on the official answer key, and whether the two answers were consistent. The answer key used for scoring included the final version, which was updated following the official appeals process that allows examinees scoring 58–65% to challenge answers with textbook-based justification, potentially resulting in multiple accepted correct answers for individual questions. Model accuracy was recorded as binary (correct/incorrect), and agreement was noted as either consistent or inconsistent across the two attempts.

In parallel, anonymized aggregate examinee performance data were provided by the Israeli Anesthesiology Examination Committee. Each question included the percentage of examinees selecting each answer option, the proportion of correct and incorrect responses, the number of unanswered items, and quartile-based performance metrics (lower and upper quartiles). These data were aligned question-by-question with model outputs to enable direct item-level comparison.

Question characteristics were stratified by format (text only vs. visual elements), difficulty (easy ≥ 70%, medium 40–69%, hard < 40% correct based on examinee performance), and discrimination ability (divided into tertiles based on the difference between upper and lower quartile examinee performance).

Study outcomes

The study outcomes included:

1. Overall accuracy for each LLM compared to human examinee performance (both average and upper quartile) across all three examinations.

2. Agreement analysis, including self-consistency between attempts and agreement with human responses.

3. Performance stratification by question format, difficulty and discrimination ability.

4. Response pattern analysis comparing first and second attempts.

5. Performance by topic areas according to Miller’s Anesthesia, 9th Edition, chapters.

Statistical methods

Statistical analyses were performed using R (version 4.5.0). Descriptive statistics included percentages with 95% confidence intervals (CI) calculated using exact binomial methods for accuracy measures. Paired t-tests were used to compare question-level accuracy between models and between each model and human performance, with effect sizes calculated using Cohen’s d to quantify the magnitude of differences. Comparisons were also performed between each LLM and the upper quartile human performance to assess whether AI could match top-performing examinees. Cohen’s kappa coefficients with 95% CI were calculated to assess self-consistency between first and second attempts for each LLM, agreement on correctness (correct/incorrect) between each LLM and human examinees, and agreement on specific answer choices between each LLM and the human majority response. Performance analyses were stratified by predetermined categories. Statistical significance was set at P < 0.05.

Results

Over the three examination periods, 381 anesthesiology examinees took the Israeli board examinations (145 examinees in 2023, 120 in 2024 season 1, and 116 in 2024 season 2). Each examination consisted of 150 MCQs with a passing threshold of 65%. A total of 450 questions across these three examinations were analyzed, with each LLM completing two independent attempts per exam. Human examinees’ pass rates across the three examinations were 52.41% (76/145) in 2023, 48.33% (58/120) in 2024 season 1, and 44.83% (52/116) in 2024 season 2, with an overall pass rate of 48.82% (186/381).

Overall accuracy

Overall accuracy across all examinations was 73.67% (95% CI: 70.66, 76.52) for Claude 3.7 Sonnet, 64.44% (95% CI: 61.22, 67.58) for ChatGPT-4, and 62.77% (95% CI: 62.42, 63.22) for human examinees. Claude 3.7 Sonnet significantly outperformed both human examinees (mean difference: 11% [95% CI; 8%, 14%], P < 0.001, Cohen’s d = 0.31) and ChatGPT-4 (mean difference: 9% [95% CI; 5%, 13%], P < 0.001, Cohen’s d = 0.22), while ChatGPT-4 performance did not significantly differ from human examinees (mean difference: 2% [95% CI; -2%, 5%], P = 0.368, Cohen’s d = 0.05). Claude 3.7 Sonnet exceeded the 65% passing threshold on all six attempts, while ChatGPT-4 achieved passing scores on only three of six attempts. Human examinee performance remained consistent throughout the three examinations (62.90%, 63.43%, and 61.98%). Both LLMs showed minimal variation between first and second attempts. Detailed accuracy performance results by exam session, attempt, and overall summary are presented in Table 1 and illustrated in Fig. 1.

Table 1.

Accuracy performance by exam session, attempt, and overall summary.

Exam	Season	Attempt	Accuracy (95% CI)
			Claude Sonnet 3.7	ChatGPT-4	Human examinees
2023	1	First	71.33 (63.39, 78.41)	60.00 (51.69, 67.90)	-
		Second	73.33 (65.51, 80.22)	68.00 (59.90, 75.37)	-
		Average	72.33 (66.90, 77.32)	64.00 (58.28, 69.44)	62.90 (62.27, 63.56)
2024	1	First	74.00 (66.21, 80.81)	64.00 (55.77, 71.67)	-
		Second	72.00 (64.09, 79.02)	64.00 (55.77, 71.67)	-
		Average	73.00 (67.60, 77.94)	64.00 (58.28, 69.44)	63.43 (62.75, 64.16)
2024	2	First	74.67 (66.93, 81.41)	64.67 (56.45, 72.29)	-
		Second	76.67 (69.07, 83.18)	66.00 (57.83, 73.53)	-
		Average	75.67 (70.40, 80.41)	65.33 (59.65, 70.71)	61.98 (61.32, 62.77)
Overall	-	-	73.67 (70.66, 76.52)	64.44 (61.22, 67.58)	62.77 (62.42, 63.22)

Values represent percentage accuracy with 95% CI in brackets. Each LLM was tested twice on each 150-question exam, with results shown for the 1st attempt, the 2nd attempt, and the average. Human performance represents aggregated results from examinees who took these official examinations.

CI confidence intervals, LLM large language models.

Fig. 1.

Accuracy performance by exam session and overall summary. Bar charts display mean accuracy percentages with 95% CI (error bars) for Claude 3.7 Sonnet (red), ChatGPT-4 (blue), and human examinees (green) across three examination sessions and overall performance. The horizontal dashed line indicates the 65% passing threshold. LLM performance represents the average of two independent testing sessions per exam. Each exam consisted of 150 MCQs. Human performance data represents aggregated anonymous results from examinees who took these official examinations. CI confidence intervals, LLM large language models, MCQ multiple-choice questions.

Notably, when compared to the upper quartile of human examinees, who achieved 78.05% accuracy (95% CI: 77.37, 78.73), both LLMs performed significantly lower. Claude 3.7 Sonnet’s performance was 4% below the upper quartile (95% CI: 1%, 8%, P = 0.01, Cohen’s d = -0.13), while ChatGPT-4 performed 14% below (95% CI: 10%, 17%, P < 0.001, Cohen’s d = -0.37).

Agreement analysis

Self-consistency between first and second attempts was substantial for both LLMs, with Claude 3.7 Sonnet achieving κ = 0.68 (95% CI: 0.61, 0.76) and ChatGPT-4 achieving κ = 0.66 (95% CI: 0.59, 0.73). Agreement on correct/incorrect classifications between LLMs and human examinees was fair to moderate, with Claude 3.7 Sonnet showing κ = 0.39 (95% CI: 0.30, 0.48) and ChatGPT-4 showing κ = 0.34 (95% CI: 0.26, 0.43). When comparing specific answer choices with human majority responses, Claude 3.7 Sonnet demonstrated substantial agreement (κ = 0.63, 95% CI: 0.57, 0.68), while ChatGPT-4 showed moderate agreement (κ = 0.50, 95% CI: 0.45, 0.56). Detailed agreement analysis between LLMs and human performance using Cohen’s kappa is presented in Table 2.

Table 2.

Agreement analysis between LLMs and human performance.

	Kappa (95% CI)
ChatGPT-4 vs. Human Examinees (Correctness)	0.34 (0.26, 0.43)
Claude 3.7 Sonnet vs. Human Examinees (Correctness)	0.39 (0.3, 0.48)
ChatGPT-4 First vs. Second Attempt	0.66 (0.59, 0.73)
Claude 3.7 Sonnet First vs. Second Attempt	0.68 (0.61, 0.76)
ChatGPT-4vs Human Majority (Answer Choice)	0.50 (0.45, 0.56)
Claude 3.7 Sonnet vs. Human Majority (Answer Choice)	0.63 (0.57, 0.68)

Cohen’s kappa coefficients with 95% CI. Correctness agreement compares binary correct/incorrect classifications. Answer choice agreement compares selected options with the human majority response. Self-consistency measured agreement between LLM attempts.

CI confidence interval, LLM large language model.

When stratified by question difficulty, Claude 3.7 Sonnet demonstrated decreasing self-consistency with increasing difficulty (κ = 0.76, 95% CI: 0.63, 0.90 for easy, κ = 0.59, 95% CI: 0.46, 0.73 for medium, κ = 0.54, 95% CI: 0.34, 0.74 for hard questions), while ChatGPT-4 showed relatively stable consistency across difficulty levels (κ = 0.58, 95% CI: 0.44, 0.73, κ = 0.63, 95% CI: 0.51, 0.75, and κ = 0.60, 95% CI: 0.41, 0.80, respectively).

Performance by question characteristics

Question characteristics were distributed as follows: for format, 394 (87.6%) were text-only and 56 (12.4%) included visual elements; by difficulty, 200 (44.4%) were easy, 177 (39.3%) were medium, and 73 (16.2%) were hard; and for discrimination ability, questions were evenly distributed across tertiles, with 150 in each tertile.

Performance analysis stratified by question format revealed notable differences between the LLMs and human residents. For text-only questions, Claude 3.7 Sonnet achieved 75.13% accuracy (95% CI: 71.95, 78.11), ChatGPT-4 achieved 65.36% (95% CI: 61.92, 68.68), and human examinees achieved 62.53% (95% CI: 62.13, 62.98). For questions containing visual elements, accuracy decreased to 63.39% (95% CI: 53.76, 72.29) for Claude 3.7 Sonnet and 58.04% (95% CI: 48.34, 67.30) for ChatGPT-4, while human examinees performed slightly better at 64.48% (95% CI: 63.56, 65.80).

Performance also varied substantially by question difficulty level. For easy questions, Claude 3.7 Sonnet achieved 86.75% accuracy (95% CI: 83.03, 89.91) compared to 81.75% (95% CI: 77.61, 85.41) for ChatGPT-4 and 81.39% (95% CI: 80.95, 81.91) for human examinees. For medium difficulty questions, accuracies were 73.45% (95% CI: 68.52, 77.97), 57.63% (95% CI: 52.29, 62.83), and 55.69% (95% CI: 55.18, 56.48), respectively. For hard questions, performance dropped markedly to 38.36% (95% CI: 30.44, 46.76) for Claude 3.7 Sonnet, 33.56% (95% CI: 25.97, 41.84) for ChatGPT-4, and 28.90% (95% CI: 28.11, 29.97) for human examinees.

When stratified by discrimination ability tertiles, both LLMs performed better on high- and medium-discrimination questions. Claude 3.7 Sonnet achieved 77.33% (95% CI: 72.17, 81.95) on high discrimination questions, 79.33% (95% CI: 74.30, 83.77) on medium, and 64.33% (95% CI: 58.63, 69.75) on low discrimination questions. ChatGPT-4 followed a similar pattern with 65.00% (95% CI: 59.31, 70.39), 66.00% (95% CI: 60.33, 71.35), and 62.33% (95% CI: 56.58, 67.84), respectively. Performance stratified by question characteristics is illustrated in Fig. 2.

Fig. 2.

Performance by question characteristics. Bar charts display mean accuracy percentages with 95% CI (error bars) for Claude 3.7 Sonnet (red), ChatGPT-4 (blue), and human examinees (green). Data are stratified by question format (text only, visual elements), difficulty (easy ≥ 70%, medium 40–69%, hard < 40%), and item discrimination (tertiles of upper–lower quartile differences). CIs are calculated using exact binomial methods appropriate for proportion data, which may appear asymmetrical due to the bounded nature of percentage data and smaller sample sizes for individual topics. The horizontal dashed line marks the 65% passing threshold. CI confidence interval.

Response pattern analysis

Analysis of response patterns between first and second attempts revealed distinct consistency profiles for each LLM. Claude 3.7 Sonnet answered both attempts correctly for 304 questions (67.6%), made consistent errors on 73 questions (16.2%), and showed inconsistent errors on 18 questions (4.0%). The model corrected itself on 26 questions (5.8%) in the second attempt and regressed on 29 (6.4%). ChatGPT-4 demonstrated a similar but slightly less consistent pattern, with both attempts correct for 255 questions (56.7%), consistent errors on 97 questions (21.6%), and inconsistent errors on 28 questions (6.2%). ChatGPT-4 showed equal rates of self-correction and regression (42 questions each, 9.3%). Response consistency patterns are visualized in Fig. 3.

Fig. 3.

Response consistency patterns between the first and second attempts. Stacked bar charts display the distribution of response consistency patterns between first and second attempts for each LLM across all 450 questions. Each segment shows the percentage of questions exhibiting that specific response pattern. LLM large language model.

Performance by topic areas

Performance varied considerably across anesthesiology topics. Among the 10 most frequently tested areas, both models achieved their highest accuracy in Cardiac Physiology, with Claude 3.7 Sonnet scoring 96.88% (95% CI: 83.78, 99.92, n = 16) and ChatGPT-4 scoring 81.25% (95% CI: 63.56, 92.79, n = 16), compared to 68.81% (95% CI: 67.82, 71.84) for human examinees. Conversely, both models showed their weakest performance in Anesthesia for Obstetrics, with Claude 3.7 Sonnet at 54.55% (95% CI: 32.2, 75.61, n = 11) and ChatGPT-4 at 50.00% (95% CI: 28.22, 71.78, n = 11), compared to 64.36% (95% CI: 61.80, 66.93) for human examinees. Human examinees demonstrated more consistent performance across topics, with the highest accuracy in Respiratory Physiology and Pathophysiology (73.13% [95% CI: 70.64, 74.69], n = 15) and the lowest in Cardiovascular Monitoring (61.45% [95% CI: 59.86, 65.05], n = 11). Detailed performance data across all topic areas is presented in Supplementary Table S1, with the top ten topics by question frequency illustrated in Fig. 4.

Fig. 4.

Performance by topic area (top ten topics by question count). Accuracy percentages with 95% CI for Claude 3.7 Sonnet (red), ChatGPT-4 (blue), and human examinees (green) across the ten most frequently tested topics. CIs are calculated using exact binomial methods appropriate for proportion data, which may appear asymmetrical due to the bounded nature of percentage data and smaller sample sizes for individual topics. The dashed vertical line indicates the 65% passing threshold. CI confidence interval.

Discussion

To the best of our knowledge, this study presents the first direct comparison between LLMs and anesthesiology examinees on official national board examinations. Claude 3.7 Sonnet achieved 73.67% accuracy, significantly outperforming both average human examinees (62.77%) and ChatGPT-4 (64.44%). However, when compared to upper quartile human performance (78.05%), both LLMs fell short, with Claude 3.7 Sonnet underperforming by 4% and ChatGPT-4 by 14%. These results underscore that while AI currently demonstrates competence exceeding that of the average examinee, top-quartile MCQ performance remains a human advantage at present, carrying profound implications for anesthesiology education, assessment, and the integration of AI into medical training.

Previous research has examined LLM performance on anesthesiology exams using practice materials from the ABA^13–15 and the FRCA^16,17, demonstrating varying performance levels, with accuracy ranging from approximately 45% to 80%. The improved performance observed in our study likely reflects both the use of more advanced AI models (Claude 3.7 Sonnet and ChatGPT-4) compared to earlier generations tested in previous research, as well as potential differences between practice materials and official examination questions. Unlike these studies, our analysis utilizes official examination questions with corresponding examinee performance data, providing a more direct assessment of current AI capabilities relative to those of human clinicians. Recent studies examining AI performance on medical board examinations across other specialties, including sleep medicine, otolaryngology, orthopedics, and urology, have reported similar performance patterns and reliability concerns, suggesting that our anesthesiology findings reflect broader patterns in AI medical examination capabilities rather than specialty-specific phenomena^19–22.

The trajectory of LLM performance across successive model generations provides important context for interpreting our findings. Evidence from anesthesiology-specific studies demonstrates a clear upward trend: ChatGPT-3.5 achieved approximately 48–56% accuracy on ABA-style practice questions^13,14, while ChatGPT-4 improved to 69–80% on similar materials^13,15,16. More recently, five current-generation LLMs, including ChatGPT-4 and Claude, all surpassed the 70% ABA BASIC passing threshold, with accuracies ranging from 86% to 92%²³. Beyond anesthesiology, this upward trend extends to other specialties: across successive GPT generations, progressive improvement from 57% (ChatGPT-3.5) to 93% (ChatGPT-5) has been demonstrated on sleep medicine board questions²⁴. However, performance on official examinations administered in non-English languages appears consistently lower than on English-language practice materials. ChatGPT-4o achieved 84% on the Chilean anesthesiology examination²⁵, while ChatGPT-4 achieved 69% and DeepSeek-R1 achieved 73% on the Chinese anesthesiology attending physician examination²⁶, and in our study, ChatGPT-4 achieved 64% on the Hebrew-language Israeli examination. These findings suggest that while performance continues to improve on English-language practice materials, a gap persists between practice and official examination performance.

The differential performance on visual versus text-based content highlights current limitations in LLM multimodal capabilities for medical applications. Both models showed reduced accuracy on questions requiring interpretation of diagnostic imaging, physiological waveforms, and clinical charts, while human examinees maintained consistent performance across both formats. This pattern suggests that effective clinical application of AI in anesthesiology will require careful consideration of task types, particularly those involving visual medical data interpretation. Given the central role of monitoring displays, imaging studies, and graphical information in anesthesiology practice, these performance differences have practical implications for determining appropriate AI assistance roles in clinical settings.

Performance stratification by question characteristics revealed important insights into AI capabilities and limitations. Both LLMs demonstrated strong performance on easy questions; however, performance gaps narrowed substantially as difficulty increased, with all performers struggling on hard questions where accuracies dropped below 40%. Performance on questions with higher discrimination ability (those that better differentiate between high and low-performing examinees) was similarly superior for both LLMs. These patterns suggest that while AI excels at pattern recognition and recall of established knowledge, questions with low human accuracy or poor discrimination present challenges for current models. However, question difficulty can arise from multiple factors, including poor wording, obscure factual content, or complex reasoning demands, and our methodology cannot distinguish between these different sources of difficulty. Additional analysis of self-consistency by difficulty level revealed that Claude 3.7 Sonnet was notably more consistent on easy questions than on hard questions, whereas ChatGPT-4 maintained more uniform consistency across difficulty levels, suggesting different underlying response generation characteristics between models. However, confidence intervals were wide, particularly for hard questions (n = 73), and these findings should be interpreted cautiously.

Topic-specific analysis revealed fundamental differences between AI and human competence. While LLMs excelled in domains with well-established principles like Cardiac Physiology, they demonstrated extreme variability across topics. Both models showed lower performance in areas such as Ambulatory (Outpatient) Anesthesia, Anesthesia for Thoracic Surgery, and Peripheral Nerve Blocks and Ultrasound Guidance for Regional Anesthesia. They similarly struggled with complex pharmacological domains such as Cerebral Physiology and the Effects of Anesthetic Drugs, and Neuromuscular Physiology and Pharmacology. In contrast, human examinees maintained more consistent performance across all domains. Such consistency across domains highlights where human assessment performance advantages persist. However, these topic-specific findings should be interpreted cautiously given the small sample sizes for individual topics and multiple comparisons without statistical adjustment.

These findings reveal significant limitations that preclude current LLM use in anesthesiology education. With Claude achieving 73.7% overall accuracy and ChatGPT-4 demonstrating consistent correct responses on only 56.7% of questions across both attempts, current LLMs pose substantial risks for knowledge acquisition in medical education. The pattern of extreme variability across topic areas, with performance ranging from near-perfect (96.8% in cardiac physiology) to concerning levels (30.0% in ambulatory anesthesia), demonstrates unreliable performance that could mislead learners. Given these reliability limitations, educators and learners should exercise considerable caution regarding AI integration in anesthesiology training. Current LLMs should not be used as primary educational resources without extensive human supervision, fact-checking, and validation mechanisms.

These results must not be conflated with clinical competence. Anesthesiology demands real-time integration of multiple physiological parameters, procedural skills, crisis management, and compassionate patient care—capabilities extending far beyond standardized testing. The consistent struggles across certain topic areas demonstrate current limitations in AI assessment performance. Board examinations, while important milestones, represent only one dimension of professional competence.

Study limitations and future directions

This study has several significant limitations that suggest directions for future research. First, we evaluated only one national examination system with specific performance characteristics (average human examinee performance of 62.77% fell below the 65% passing threshold), which may differ from board examination pass rates and performance patterns in other countries; multi-national studies would enhance generalizability. Second, the MCQ format provides answers among defined options, potentially inflating AI performance; while recent research by Blacker et al. has begun investigating AI performance on oral examinations²⁷, showing comparable module ratings but subjective inferiority due to response length and focus issues, further research on open-ended questions would provide additional insights into AI knowledge assessment capabilities. Similarly, evaluating LLM performance on the Israeli Step 2 oral board examination would further clarify AI capabilities in clinical reasoning and decision-making contexts. Third, aggregate human performance data precluded individual-level comparisons; future studies with access to individual examinee data could provide more detailed insights into performance patterns and enable direct question-by-question comparisons between individual examinees and AI models. Fourth, our findings represent a temporal snapshot of rapidly evolving AI technology, and the lower performance observed on non-English official examinations suggests that language may independently influence LLM accuracy. Ongoing benchmarking across languages and model generations remains essential. Fifth, this study focused on two widely used LLMs (Claude 3.7 Sonnet and ChatGPT-4); evaluation of additional models such as Copilot Pro, Gemini Advanced, and others remains an important area for future research to provide broader insights into AI performance on medical examinations. Finally, while examinees knew exam questions were based primarily on Miller’s Anesthesia 9th Edition, LLMs lacked access to this reference due to context window limitations and copyright considerations. Instead, LLMs drew from their broader medical training data. Future research could obtain publisher permission or utilize platforms like Google’s NotebookLM that can process larger reference materials while respecting copyright restrictions.

Conclusions

This study demonstrates that advanced LLMs have currently achieved performance levels exceeding average human examinees on anesthesiology board examinations, though they presently remain below the performance of the best-performing examinees. However, superior test performance should not overshadow the multifaceted nature of anesthetic practice. Given demonstrated reliability limitations, medical educators must ensure robust oversight of any AI educational applications. As AI technology continues to advance, the goal should not be to achieve AI superiority but to leverage these tools to enhance human performance, improve educational outcomes, and ultimately deliver better patient care through the synergistic combination of AI and human performance advantages.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

ABA

American Board of Anesthesiology

AI

Artificial intelligence

CI

Confidence intervals

FRCA

Fellowship of the Royal College of Anaesthetists

LLMs

Large Language Models

MCQ

Multiple-choice questions

Author contributions

All authors reviewed the manuscript.

Funding

None declared.

Data availability

The data of this study is available from the corresponding author upon reasonable request.

Declarations

Competing interests

The authors declare no competing interests.