Skip to main content
NCBI home page
Journal of Medical Internet Research logoLink to Journal of Medical Internet Research
. 2025 Apr 30;27:e64486. doi: 10.2196/64486

Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis

Ling Wang 1,2,#, Jinglin Li 2,#, Boyang Zhuang 3,#, Shasha Huang 4,#, Meilin Fang 2,#, Cunze Wang 2, Wen Li 1, Mohan Zhang 2, Shurong Gong 5,✉
Editor: Xiaomeng Ma
Reviewed by: Mahshad Lotfinia, Xiao-Meng Zhang
PMCID: PMC12079073  PMID: 40305085

Abstract

Background

Large language models (LLMs) have flourished and gradually become an important research and application direction in the medical field. However, due to the high degree of specialization, complexity, and specificity of medicine, which results in extremely high accuracy requirements, controversy remains about whether LLMs can be used in the medical field. More studies have evaluated the performance of various types of LLMs in medicine, but the conclusions are inconsistent.

Objective

This study uses a network meta-analysis (NMA) to assess the accuracy of LLMs when answering clinical research questions to provide high-level evidence-based evidence for its future development and application in the medical field.

Methods

In this systematic review and NMA, we searched PubMed, Embase, Web of Science, and Scopus from inception until October 14, 2024. Studies on the accuracy of LLMs when answering clinical research questions were included and screened by reading published reports. The systematic review and NMA were conducted to compare the accuracy of different LLMs when answering clinical research questions, including objective questions, open-ended questions, top 1 diagnosis, top 3 diagnosis, top 5 diagnosis, and triage and classification. The NMA was performed using Bayesian frequency theory methods. Indirect intercomparisons between programs were performed using a grading scale. A larger surface under the cumulative ranking curve (SUCRA) value indicates a higher ranking of the corresponding LLM accuracy.

Results

The systematic review and NMA examined 168 articles encompassing 35,896 questions and 3063 clinical cases. Of the 168 studies, 40 (23.8%) were considered to have a low risk of bias, 128 (76.2%) had a moderate risk, and none were rated as having a high risk. ChatGPT-4o (SUCRA=0.9207) demonstrated strong performance in terms of accuracy for objective questions, followed by Aeyeconsult (SUCRA=0.9187) and ChatGPT-4 (SUCRA=0.8087). ChatGPT-4 (SUCRA=0.8708) excelled at answering open-ended questions. In terms of accuracy for top 1 diagnosis and top 3 diagnosis of clinical cases, human experts (SUCRA=0.9001 and SUCRA=0.7126, respectively) ranked the highest, while Claude 3 Opus (SUCRA=0.9672) performed well at the top 5 diagnosis. Gemini (SUCRA=0.9649) had the highest rated SUCRA value for accuracy in the area of triage and classification.

Conclusions

Our study indicates that ChatGPT-4o has an advantage when answering objective questions. For open-ended questions, ChatGPT-4 may be more credible. Humans are more accurate at the top 1 diagnosis and top 3 diagnosis. Claude 3 Opus performs better at the top 5 diagnosis, while for triage and classification, Gemini is more advantageous. This analysis offers valuable insights for clinicians and medical practitioners, empowering them to effectively leverage LLMs for improved decision-making in learning, diagnosis, and management of various clinical scenarios.

Trial Registration

PROSPERO CRD42024558245; https://www.crd.york.ac.uk/PROSPERO/view/CRD42024558245

Keywords: large language models, LLM, clinical research questions, accuracy, network meta-analysis, PRISMA

Introduction

Recent research has demonstrated the considerable success of large language models (LLMs) in a multitude of natural language tasks, including automatic summarization (the generation of a condensed version of a passage of text), machine translation (the automatic translation of text from one language to another), and question-and-answer systems (the construction of a system to automatically answer questions based on a passage of text) [1]. In this context, with the development of big biomedical data and artificial intelligence, the emergence of flexible natural language processing models such as ChatGPT provides a number of new possibilities for health care and biomedical research and has the potential to be a turning point in the field [2-4].

Although LLMs have shown great potential in the medical field, medicine is a demanding field, it is associated with life, and its complexity as well as specificity mean that any application must meet extremely high standards of accuracy. Controversy remains about whether LLMs can be applied to the medical field. Mu and He [5] reviewed the potential applications and challenges of ChatGPT in health care, noting that a lack of understanding of medical knowledge and specialized medical backgrounds hinder the ability of ChatGPT to delve into the complexity of medical concepts and terminology. Consequently, the capacity of ChatGPT to address specific medical queries, diagnose ailments, or furnish precise medical recommendations is restricted. Another study noted that the role of LLMs in health care may be limited by the presence of bias in training materials, their tendency to “hallucinate,” and ethical and legal considerations when LLMs provide inaccurate advice that leads to patient harm, as well as patient privacy issues [6].

Given the controversy over the application of LLMs in medicine and the continuous emergence and versioning of LLMs, more research has been devoted to evaluating the performance of various LLMs in medicine to provide stronger evidence. In addition to ChatGPT developed by OpenAI, the performance of many other LLMs such as Microsoft (eg, Copilot [7]), Google (eg, Gemini [8]), and Meta (eg, LLaMA [9]) in the medical domain has also been compared. Many aspects of assessment have been included, such as medical exams [10], case text diagnosis [11], and disease classification or grading [12].

Unfortunately, there are differences in the performance of different LLMs in different studies. For example, in a study by Vaishya et al [13] that explored the performance of ChatGPT-3.5, ChatGPT-4, and Google Bard when answering 120 multiple-choice questions, the results showed that Google Bard had 100% accuracy and was significantly more accurate than both ChatGPT-3.5 and ChatGPT-4 (P<.001). Another study showed that ChatGPT-4 was more accurate than Google Bard (83% vs 76%) [14]. At present, most related research is limited to a single type of LLM [15,16] or a specific domain area [17,18], and there is no high-level evidence comparing the accuracy rankings of different LLMs when responding to clinical research questions.

Therefore, this study aimed to compare the accuracy of different LLMs when answering clinical research questions, including objective questions, open-ended questions, top 1 diagnosis, top 3 diagnosis, top 5 diagnosis, and triage and classification. This study aimed to provide high-level evidence-based support for future clinical applications, enabling clinical workers to better use LLMs to make more accurate and informed decisions for future learning, diagnosis, and different clinical scenarios.

Methods

Network Meta-Analysis

The network meta-analysis (NMA) was based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) reporting guidelines. The PRISMA checklist is shown in Multimedia Appendix 1. The Bayesian approach permits the indirect comparison of performance between a range of LLMs that were not explicitly articulated throughout the experiment. The study protocol was defined and registered in the PROSPERO database prior to the commencement of the study.

Search Strategy and Selection Criteria

A computer search of the PubMed, Embase, Web of Science, and Scopus databases was conducted to identify relevant studies on the accuracy of different LLMs when answering questions in the medical field. The last search was updated to October 14, 2024, to identify studies published since the first search, with no restrictions on the type of study. When the results of a study were reported in multiple publications, we included the study with the richest and most recent findings. We also searched the list of literature on LLMs in medicine-related systematic reviews and manually searched the references included in the reviews for additional access to relevant literature. The search subject terms were “LLM,” “generative AI,” “open AI,” “Large language model,” “ChatGPT-3.5,” “ChatGPT-4,” “Google Bard,” and “Bing,” without any language restriction. The complete search strategies for all databases are shown in Multimedia Appendix 2.

A combination of EndNote X9 deduplication and manual deduplication was used to screen the literature in accordance with the developed inclusion criteria. The results of the literature searches conducted in different databases were then combined to create a new information database, which could be downloaded in full text. Independent review and assessment of the titles, abstracts, and full texts of the relevant literature were undertaken by 4 authors (LW, JL, BZ, and SH). The review encompassed studies using disparate LLMs systems to respond to medical queries. Letters, conference abstracts, editorials, reviews, and expert opinions for which no information was available were excluded from the review. In addition, the following studies were excluded: those that evaluated the performance of only 1 LLM; those that assessed the performance of 2 or more LLMs without specifying the LLM versions used (eg, the article only mentioned evaluating ChatGPT without mentioning ChatGPT-3.5, ChatGPT-4, or other versions), with the updated versions and timelines of various LLMs so far shown in Multimedia Appendix 3; those that assessed the performance of 2 or more LLMs but did not provide data isolating their accuracy when answering different types of questions; and the questions included in the study contained images. In addition, to reduce bias, we excluded research on accessing LLMs through an application programming interface (API).

Assessment of Results

The primary outcomes were the accuracy of LLMs when answering medical questions. These included objective questions, open-ended questions, top 1 diagnosis, top 3 diagnosis, top 5 diagnosis, and triage and classification accuracy. Objective questions are exam questions with a clear, quantifiable answer that is usually predetermined, unique, or with a limited number of options. Open-ended questions are a type of question that does not have a fixed answer nor standardized answer. Diagnosis and triage and classification are open-ended questions, but most diagnostic questions end with “What is the most probable diagnosis?” whereas triage and classification questions end with “How would you classify this disease?” Corresponding examples are shown in Multimedia Appendix 4.

Accuracy for objective questions was calculated as the number of correctly answered questions divided by the total number of questions. For diagnosis and classification, accuracy was defined as the number of cases correctly diagnosed or triaged divided by the total number of cases. Specifically for open-ended questions, accuracy was determined based on the number of questions rated “good” or “accurate” on the accuracy scale divided by the total number of questions.

Data Extraction

The 4 researchers jointly extracted and verified the following data: (1) basic information about the included studies, such as study title and first author; (2) baseline characteristics and interventions of the study population; (3) key elements evaluated for risk of bias; and (4) outcome indicators and relevant outcome measure data. Our study involved extracting raw data from each study. In cases of disagreement, these were resolved through discussion and consultation with a third party.

Quality Assessment

Because they were cross-sectional studies, the quality of the included studies was evaluated using the Newcastle-Ottawa Scale [19]. The quality assessment was conducted by 3 independent researchers (LW, JL, and BZ), with a fourth researcher (SH) resolving any disagreements. A low overall risk of bias was determined when the Newcastle-Ottawa Scale score ranged from 7 to 9, moderate risk was determined when the score was between 4 and 6, and high risk was determined when the score was 0 to 3.

Statistical Analyses

Statistical analyses were performed using Stata 18.0 and R (version 4.3.1), with the odds ratio (OR) as the analytical statistic. Accuracy was assessed using 95% CIs and the credible interval. NMA analyses were performed on different types of LLMs.

The confidence of the NMA results estimates was assessed according to the Confidence in Network Meta-Analysis (CINEMA) methodology, which is broadly based on the Grading of Recommendations Assessment, Development, and Evaluation (GRADE). An NMA was conducted within a Bayesian framework using Markov chain Monte Carlo methods and was computed using the BUGSnet and GeMTC packages in R (V.4.3.1) software. A network graph was constructed for each LLM included in the experiment in order to facilitate a comparison of the performance of multiple LLMs. The consistency between direct and indirect evidence was evaluated using a node-splitting method when there was a closed loop. If the P value between the direct, indirect, and network comparisons of the 2 interventions was >.05, we concluded that there was no statistical difference and consistency was good. The convergence of the network models derived from the Markov chain Monte Carlo simulations was assessed using trace and density plots. We used noninformative priors for all parameters and assumed common heterogeneity. Furthermore, for all LLMs, we determined the ranking probabilities, which were articulated as the surface under the cumulative ranking curve (SUCRA). Higher SUCRA values suggest superior accuracy in model ranking.

Results

Literature Search and Selection

A bibliographic search yielded 59,075 citations, of which 21,156 studies were identified as potential conditions based on abstract screening and retrieved for full text evaluation. Manual reading of the titles and abstracts of the remaining literature excluded 20,814 papers whose topics and interventions did not match the inclusion criteria for this study. Further reading of the full texts excluded the following: 174 articles that could not be separated nor extracted from the ending; 147 articles in which we were unable to separate outcome data, unable to extract outcome data, or detected issues related to images; 12 articles with unclear versions of the LLMs; and 8 articles that used an API to access LLMs. In addition, the full text of 7 articles was not available, resulting in the final inclusion of 168 articles from the literature. The literature screening process is shown in Figure 1.

Figure 1.

Figure 1

Open in a new tab

Literature screening flowchart. API: application programming interface.

Basic Characteristics of the Incorporated Literature

To assess the accuracy of different LLMs when answering medical questions, a total of 168 studies underwent a screening process to determine their suitability for inclusion. A total of 35,896 questions and 3063 clinical cases were included in the study. The basic information of the 168 studies is presented in Multimedia Appendix 5.

Quality Assessment of the Included Studies

In the quality assessment, 40 (40/168, 23.8%) studies were assessed as having a low overall risk of bias, while 128 (128/168, 76.2%) had a moderate overall risk of bias. No studies were identified as having a high overall risk of bias. The detailed quality assessment results for each study can be found in Multimedia Appendix 6.

Network Meta-Analysis

Objective Questions

The accuracy of LLMs when answering objective questions was reported in 105 studies [10,13,14,20-121]. The evidence network relationships are plotted in Figure 2A and involve 30 LLMs and a total of 33,838 multiple choice questions. Direct and indirect comparisons were formed for each LLM, partially forming a closed loop. The results of the indirect comparison are shown in Figure 3 and Multimedia Appendix 7. The red cells indicate there are statistically significant differences between the column-defining regimen and the row-defining regimen. The values in the green and blue cells are the logOR and 95% CI, respectively, from the comparison of the LLMs represented in the columns with the LLMs represented in the rows. A logOR value <0 indicates that the accuracy of the LLM corresponding to a column is lower than the LLM corresponding to a row. A value >0 indicates a higher accuracy. There was no evidence of statistically significant inconsistency (all P>.05) in the node-splitting test for NMA, except for Claude 2 versus ChatGPT-4 (P=.04), Bing chat versus people (P=.004), and Perplexity versus people (P=.04; Multimedia Appendix 8). The convergence of iterations was evaluated as good in trace and density plots, with the bandwidth tending toward 0 and reaching stability (Multimedia Appendix 9). The best probability ranking showed that ChatGPT-4o (SUCRA=0.9207) ranked first in terms of accuracy when answering objective questions, Aeyeconsult (SUCRA=0.9187) ranked second, and ChatGPT-4 (SUCRA=0.8087) ranked third (Table 1, Figure 4A).

Figure 2.

Figure 2

Open in a new tab

Comparison network diagram of different outcomes, where larger nodes indicate more questions and thicker line segments indicate more questions between 2 types of large language models (LLMs) when answering (A) objective questions, (B) open-ended questions, (C) a top 1 diagnosis, (D) a top 3 diagnosis, (E) a top 5 diagnosis, and (F) triage and classification questions.

Figure 3.

Figure 3

Open in a new tab

Indirect comparison of the accuracy of large language models (LLMs) when answering objective questions: A: instructGPT; A1: LLaMA 2; B: GTP-3; B1: LLaMA 3; C: ChatGPT-3.5; D: ChatGPT-4; D1: Mistral Large; E: ChatGPT-4o; E1: people; F1: chatENT; G: Bard; G1: ChatSonic; H: PaLM2; H1: Aeyeconsult; I: Gemini; I1: Med-PaLM 2; K: Gemini 1.5 pro; L: Bing chat; M: Copilot; N: Perplexity; O: Perplexity Pro; P: Claude; Q: Claude-instant; R: Claude 2; T: Claude 3 Opus; U: Claude 3 Sonnet; W: LLaMA 7B; X: LLaMA 13B; Y: LLaMA 33B; Z: LLaMA 65B.

Table 1.

Bayesian ranking results (surface under the cumulative ranking curve [SUCRA] value) of the network meta-analysis for each large language model (LLM).

LLM SUCRA

Objective questions Open-ended questions Top 1 diagnosis Top 3 diagnosis Top 5 diagnosis Triage and classification
instructGPT (A) 0.7805 —a — — — —
LLaMA 2 (A1) 0.2086 0.4629 0.1395 — — —
GTP-3 (B) 0.7704 — — — — —
LLaMA 3 (B1) 0.239 — — — 0.7405 —
ChatGPT-3.5 (C) 0.4343 0.5548 0.5039 0.565 0.5084 0.2093
Mixtral-8x7B (C1) — 0.6224 — — — —
ChatGPT-4 (D) 0.8087 0.8708 0.693 0.6302 0.8089 0.6185
Mistral Large (D1) 0.3842 — — — — —
ChatGPT-4o (E) 0.9207 — — — — —
People (E1) 0.6172 0.6067 0.9001 0.7126 0.6241 0.4934
chatENT (F1) 0.7687 — — — — —
Bard (G) 0.4443 0.3512 0.3353 0.4329 0.0722 0.5885
ChatSonic (G1) 0.4617 — — — — —
PaLM2 (H) 0.421 0.312 0.4496 — — 0.5197
Aeyeconsult (H1) 0.9187 — — — — —
Gemini (I) 0.4543 0.6703 0.2812 — 0.2405 0.9649
Med-PaLM 2 (I1) 0.3919 — — — — —
OcularBERT (J1) — 0.0176 — — — —
Gemini 1.5 pro (K) 0.2449 — — — 0.7905 —
Doctor GPT (K1) — 0.745 — — — —
Bing chat (L) 0.728 0.23 0.2073 0.4499 0.2042 0.3391
Docs-GPT Beta (L1) — 0.212 — — — —
Copilot (M) 0.7038 — 0.5048 — 0.2633 —
WebMD (M1) — — 0.7511 0.1452 — 0.4348
Perplexity (N) 0.4424 — 0.3980 0.4367 0.2801 —
Ada Health (N1) — — 0.8363 0.6273 — 0.3319
Perplexity Pro (O) 0.3821 — — — — —
Claude (P) 0.5048 — — — — —
Claude-instant (Q) 0.4949 — — — — —
Claude 2 (R) 0.4928 0.5647 — — — —
Claude 3 Opus (T) 0.7365 — — — 0.9672 —
Claude 3 Sonnet (U) 0.5094 — — — — —
LLaMA 7B (W) 0.1131 — — — — —
LLaMA 13B (X) 0.1365 — — — — —
LLaMA 33B (Y) 0.2147 — — — — —
LLaMA 65B (Z) 0.2721 — — — — —
Open in a new tab

aNot applicable because the LLM was not in the network.

Figure 4.

Figure 4

Open in a new tab

Surface under the cumulative ranking curve (SUCRAs) for the accuracy, with higher rankings associated with larger outcome values, of different large language models (LLMs) when answering (A) objective questions, (B) open-ended questions, (C) the top 1 diagnosis, (D) the top 3 diagnosis, (E) the top 5 diagnosis, and (F) triage and classification questions. The letters in the keys indicate the following LLMs: A: instructGPT; A1: LLaMA 2; B: GTP-3; B1: LLaMA 3; C: ChatGPT-3.5; C1: Mixtral-8x7B; D: ChatGPT-4; D1: Mistral Large; E: ChatGPT-4o; E1: people; F1: chatENT; G: Bard; G1: ChatSonic; H: PaLM2; H1: Aeyeconsult; I: Gemini; I1: Med-PaLM 2; J1: OcularBERT; K: Gemini 1.5 pro; K1: Doctor GPT; L: Bing chat; L1: Docs-GPT Beta; M: Copilot; M1: WebMD; N: Perplexity; N1: Ada Health; O: Perplexity Pro; P: Claude; Q: Claude-instant; R: Claude 2; S: Claude 2.1; T: Claude 3 Opus; U: Claude 3 Sonnet; W: LLaMA 7B; X: LLaMA 13B; Y: LLaMA 33B; Z: LLaMA 65B.

Subgroup Analysis

We stratified the results based on the fields of the problem (Multimedia Appendix 10). Based on the results, we compared the accuracy of LLMs in 6 fields: ophthalmology, orthopedics, urology, dentistry, oncology, and radiology. In ophthalmology, the LLM with the highest accuracy was Aeyeconsult (SUCRA=0.8334), followed by ChatGPT-4 (SUCRA=0.6331) and PaLM2 (SUCRA=0.5517). In the field of orthopedics, the LLM accuracy rates, from highest to lowest, were for Bard (SUCRA=0.7219), people (SUCRA=0.6802), and Bing chat (SUCRA=0.4732). For urology, Bing chat (SUCRA=0.7905) was the most accurate, followed by people (SUCRA=0.6587) and ChatGPT-4 (SUCRA=0.5941). In dentistry, ChatGPT-4 (SUCRA=0.9473) was the most accurate, followed by Bard (SUCRA=0.7068) and Gemini (SUCRA=0.5535). ChatGPT-4 (SUCRA=0.9002) performed the best in oncology, followed by ChatGPT-4o (SUCRA=0.8998) and Claude (SUCRA=0.7159). In radiology, ChatGPT-4o (SUCRA=0.9053) performed the best, ChatGPT-4 (SUCRA=0.7777) was second, and Claude 3 Opus (SUCRA=0.6935) ranked third. The SUCRAs are shown in Figure 5.

Figure 5.

Figure 5

Open in a new tab

Surface under the cumulative ranking curve (SUCRAs) for the accuracy, with higher rankings associated with larger outcome values, of different large language models (LLMs) in (A) ophthalmology, (B) orthopedics, (C) urology, (D) dentistry, (E) oncology, and (F) radiology. The letters in the keys indicate the following LLMs: C: ChatGPT-3.5; D: ChatGPT-4; E: ChatGPT-4o; E1=people; G: Bard; H: PaLM2; H1: Aeyeconsult; I: Gemini; L: Bing chat; P: Claude; T: Claude 3 Opus; U: Claude 3 Sonnet; W: LLaMA 7B; X: LLaMA 13B; Y: LLaMA 33B; Z: LLaMA 65B.

Open-Ended Questions

The accuracy of the LLMs when responding to open-ended questions was examined in 34 studies [122-155]. The relationships within the evidence network are plotted in Figure 2B and include 14 LLMs and a total of 2026 open-ended questions. Direct and indirect comparisons were formed for each LLM, partially forming a closed loop. The results of the indirect comparison are presented in Multimedia Appendix 10, where red cells indicate statistically significant differences between the column-defining regimen and the row-defining regimen (Multimedia Appendix 7). There was no evidence of a statistically significant inconsistency (all P>.05) in the node-splitting test for the NMA, except for Bard versus ChatGPT-3.5 (P=.02; Multimedia Appendix 8). The trace and density plots are shown in Multimedia Appendix 9, and from the results, the iterative convergence was good. The best probability ranking indicated that ChatGPT-4 (SUCRA=0.8708) exhibited the highest accuracy when answering open-ended questions, followed by Claude 2.1 (SUCRA=0.7796) and Doctor GPT (SUCRA=0.7450; Table 1, Figure 4B).

Top 1 Diagnosis, Top 3 Diagnosis, and Top 5 Diagnosis

The accuracy of the top 1 diagnosis in clinical cases by LLMs was reported in 19 studies [11,156-173]. The evidence network relationship diagram is shown in Figure 2C and involves 12 LLMs and a total of 1266 clinical cases. The accuracy of LLMs for the top 3 diagnosis was reported in 7 studies [158,161,169,171,174-176]. The evidence network relationships are plotted in Figure 2D and involve 8 LLMs and a total of 453 clinical cases. The accuracy of LLMs for the top 5 diagnosis in clinical cases was reported in 7 studies [158,167,168,173,177-179]. The evidence network relationships are plotted in Figure 2E and involve 11 LLMs and a total of 443 clinical cases. Each LLM formed direct and indirect comparisons, partially closing the loop.

In terms of the top 1 diagnosis and top 5 diagnosis, the results of the indirect comparison are presented in Multimedia Appendix 7, where red cells indicate statistically significant differences between the column-defining regimen and the row-defining regimen. For the top 3 diagnosis, there was no statistical difference (all P>.05) in the comparisons between the LLMs (Multimedia Appendix 7). There was no evidence of a statistically significant inconsistency (all P>.05) for the top 1 diagnosis, except for Ada Health versus ChatGPT-3.5 (P=.04). For the top 3 diagnosis and top 5 diagnosis, all P were >.05 in the node-splitting test for the NMA (Multimedia Appendix 8). Iterative convergence was good, as shown by the trace and density plots (Multimedia Appendix 9). The best probability ranking showed that, in terms of accuracy of the top 1 diagnosis in clinical cases, people ranked first (SUCRA=0.9001), Ada Health ranked second (SUCRA=0.8363), and WebMD ranked third (SUCRA=0.7511; Table 1, Figure 4C). In terms of the accuracy of the top 3 diagnosis, people ranked first (SUCRA=0.7126), ChatGPT-4 ranked second (SUCRA=0.6302), and Ada Health ranked third (SUCRA=0.6273; Table 1, Figure 4D). For the accuracy of the top 5 diagnosis, Claude 3 Opus ranked first (SUCRA=0.9672), ChatGPT-4 ranked second (SUCRA=0.8089), and Gemini 1.5 pro ranked third (SUCRA=0.7905; Table 1, Figure 4E).

Triage and Classification

The accuracy of LLMs in triage and classification was reported in 7 studies [12,167,169,174,180-182]. The evidence network relationships are plotted in Figure 2F and involve 9 LLMs and a total of 901 clinical cases. Each LLM formed direct and indirect comparisons, partially closing the loop. The results of the indirect comparison are shown in Multimedia Appendix 7. There were significant differences between Gemini and ChatGPT-3.5, ChatGPT-4, or Bing chat (P<.05). There was no evidence of a statistically significant inconsistency (all P>.05) in the node-splitting test for the NMA, except for ChatGPT-3.5 versus ChatGPT-4 (P=.045; Multimedia Appendix 8). Iterative convergence was good, as shown by the trace and density plots (Multimedia Appendix 9). The best probability ranking showed that, for the accuracy of triage and classification, Gemini ranked first (SUCRA=0.9649), ChatGPT-4 ranked second (SUCRA=0.6185), and Bard ranked third (SUCRA=0.5885), as shown in Table 1 and Figure 4F.

Discussion

Principal Findings

This study presents the most comprehensive meta-analysis to date on the accuracy of various LLMs when responding to medical queries, encompassing objective questions, open-ended questions, top 1 diagnosis, top 3 diagnosis, top 5 diagnosis, and triage and classification. Variations in accuracy among different LLMs were observed. ChatGPT-4o demonstrated the highest accuracy when answering objective questions, while ChatGPT-4 excelled at open-ended questions. The superior performance of people at the top 1 diagnosis and top 3 diagnosis suggests that human expertise is generally more dependable than LLMs in complex medical scenarios, while Claude 3 Opus seems to perform the best in the top 5 diagnosis. In terms of triage and classification, Gemini appeared to be more reliable.

In addition, we stratified LLMs according to the medical field in which the objective questions were located and explored their accuracy in 6 fields: ophthalmology, orthopedics, urology, dentistry, oncology, and radiology. We found that Aeyeconsult performed the best in ophthalmology, Bard performed the best in orthopedics, Bing chat performed the best in urology, ChatGPT-4 performed the best in both dentistry and oncology, and ChatGPT-4o had the highest accuracy in radiology.

At present, language models based on transformer architecture, whether pretrained or fine-tuned using biomedical corpora, have been proven effective in a series of natural language processing benchmarks in the biomedical field [183]. We attempted to analyze the reasons for the performance differences when different LLMs answer questions. Parameter size is an important factor affecting the accuracy of LLMs when answering questions. Research has found that, when the parameter size of the PaLM model is expanded from 8B to 40B, the accuracy of answering medical questions is doubled [184]. However, the practicality of a model depends not only on its number of parameters but also on many factors such as its training data and architecture, fine-tuning protocols, and overall architecture [185]. Taking GPT-4 as an example, it achieved a higher performance than its predecessor by adopting more advanced training data and architecture. The timeliness and accuracy of training data are also crucial for model performance. Today, models can not only rely on a limited set of pretraining data but also obtain the latest knowledge from the internet in real time. For example, Bing AI and Google Bard already have the ability to obtain real-time updates, and ChatGPT has also begun to follow suit by accepting plugins to expand its capabilities [185,186].

In addition, we found that some models fine-tuned on the backend LLM can achieve higher accuracy and less energy consumption in specific fields. For example, in the field of ophthalmology, Aeyeconsult integrates many ophthalmic data sets based on GPT-4 for training and generation [24]. This targeted training can significantly improve its performance in ophthalmic clinical tasks. Other possible data sources include clinical texts and accurate medical information, such as guidelines and peer-reviewed literature. In fact, there are already some models built or fine-tuned based on clinical text, such as SkinGPT-4 and ChatDoctor, which perform better overall than various general LLMs at biomedical natural language processing tasks [187,188].

Progress on various grand prognostic models has been very rapid, with a newer, more arithmetically powerful version being released every few months. However, our results show that the newer versions do not necessarily outperform the older ones in terms of performance when measured as accuracy, possibly because the newer versions incorporate fewer studies, which may have biased the results somewhat. In addition, updated versions such as ChatGPT-4V provided multimodal models (eg, that can evaluate image problems), and these models may have a greater advantage for image evaluation, for example.

Studies indicate LLMs outperform humans at exams like medical licensing, orthopedics, and pediatrics globally, highlighting LLMs’ potential as a study aid. For the top 1 diagnosis and top 3 diagnosis, human accuracy is higher than that of LLMs. Despite the fact that Claude 3 Opus outperformed humans in the top 5 diagnostic results, due to the high level of accuracy required in the medical field and the multifaceted information and complex decision-making involved in medical diagnosis, we still recommend that LLMs should only be used as an auxiliary tool to assist doctors with more efficient data analysis and preliminary diagnostic recommendations.

Several meta-analyses have been conducted to assess the accuracy of LLMs in health care [15,189,190]. However, it is very unfortunate that the LLMs included in these studies included ChatGPT only and that some of the studies simply evaluated its performance on exams. Some studies did not differentiate between the types of questions answered by ChatGPT, which led to a significant amount of heterogeneity between the studies, resulting in biased results.

We acknowledge certain limitations in our study. First, for the top 3 diagnosis, top 5 diagnosis, and triage and classification, this may bias the results due to the number of included studies as well as the sample size, so caution is needed when interpreting these results. Although we minimized the heterogeneity of the research as much as possible, we cannot deny that the inclusion of different fields of study and the complexity of LLMs (such as different instructions and questioning dates) can affect the results of the study and generate heterogeneity. Therefore, caution should be exercised when interpreting the results. In addition, we did not assess the accuracy of multimodal grand prognostic models when solving medical image–related problems; with the development of artificial intelligence, more multimodal models are being developed, and in the future, these models will become indispensable in the exploration of image-based problems in the medical field.

Conclusion

Existing studies suggest that ChatGPT-4o has an advantage for answering objective questions. For open-ended questions, ChatGPT-4 may be more credible. Humans are more accurate in the top 1 diagnosis and top 3 diagnosis of clinical cases. Claude 3 Opus performs better in the top 5 diagnosis, while for classification accuracy, Gemini is more advantageous. Although some LLMs excel at addressing medical queries, caution is advised due to the critical need for precision and rigor in medicine. Future high-quality studies and trials are necessary to gather more scientific evidence.

Acknowledgments

This work was supported by the Training Program for Young and Middle-aged Backbone Talents of Fujian Provincial Health Commission (grant number 2022GGA001), Natural Science Foundation of Fujian Province (grant number 2021J01395 and 2024J011032), Foundations of Department of Finance of Fujian Province (grant number Min Cai Zhi (2023) 830 and Min Cai Zhi (2024) 881), Joint Funding Projects for Innovation in Science and Technology of Fujian Province (grant number 2023Y9330), and Internal Supporting Project of Fuzhou University Affiliated Provincial Hospital (grant number 0080072220).

Abbreviations

API

application programming interface

CINEMA

Confidence in Network Meta-Analysis

GRADE

Grading of Recommendations Assessment, Development, and Evaluation

LLM

large language model

NMA

network meta-analysis

OR

odds ratio

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

SUCRA

surface under the cumulative ranking curve

Multimedia Appendix 1

PRISMA checklist.

Multimedia Appendix 2

Search strategy.

Multimedia Appendix 3

Versions and timelines of LLMs iterations.

Multimedia Appendix 4

Examples of the outcomes.

Multimedia Appendix 5

Description of 168 studies included.

jmir_v27i1e64486_app5.docx (195.9KB, docx)
Multimedia Appendix 6

Quality assessment of observational study.

Multimedia Appendix 7

Indirect comparison results.

Multimedia Appendix 8

Node splitting inconsistency test.

Multimedia Appendix 9

Trace and density plots.

Multimedia Appendix 10

Objective questions are stratified according to different fields of the questions.

Data Availability

The data sets generated or analyzed during this study are available from the corresponding author on reasonable request.

Footnotes

Authors' Contributions: All authors were involved in the conceptualization and design of the study and reviewed all documents and materials. LW, JL, BZ, and SH collected the data, performed data analysis, interpreted the results, and wrote the first draft of the manuscript. CW, WL, and MZ were involved in the development of the protocol for the systematic review and critically reviewed the results and the manuscript. MF and SG were involved in the development of the protocol and revised the manuscript. All authors read and approved the final manuscript.

Conflicts of Interest: None declared.

References

  • 1.Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, Moy L. ChatGPT and other large language models are double-edged swords. Radiology. 2023 Apr;307(2):e230163. doi: 10.1148/radiol.230163. [DOI] [PubMed] [Google Scholar]
  • 2.No authors listed Will ChatGPT transform healthcare? Nat Med. 2023 Mar 14;29(3):505–506. doi: 10.1038/s41591-023-02289-5.10.1038/s41591-023-02289-5 [DOI] [PubMed] [Google Scholar]
  • 3.Park SH, Pinto-Powell R, Thesen T, Lindqwister A, Levy J, Chacko R, Gonzalez D, Bridges C, Schwendt A, Byrum T, Fong J, Shasavari S, Hassanpour S. Preparing healthcare leaders of the digital age with an integrative artificial intelligence curriculum: a pilot study. Med Educ Online. 2024 Dec 31;29(1):2315684. doi: 10.1080/10872981.2024.2315684. https://www.tandfonline.com/doi/10.1080/10872981.2024.2315684?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Sblendorio E, Dentamaro V, Lo Cascio A, Germini F, Piredda M, Cicolini G. Integrating human expertise and automated methods for a dynamic and multi-parametric evaluation of large language models' feasibility in clinical decision-making. Int J Med Inform. 2024 Aug;188:105501. doi: 10.1016/j.ijmedinf.2024.105501. https://linkinghub.elsevier.com/retrieve/pii/S1386-5056(24)00164-3 .S1386-5056(24)00164-3 [DOI] [PubMed] [Google Scholar]
  • 5.Mu Y, He D. The potential applications and challenges of ChatGPT in the medical field. IJGM. 2024 Mar;Volume 17:817–826. doi: 10.2147/ijgm.s456659. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Park Y, Pillai A, Deng J, Guo E, Gupta M, Paget M, Naugler C. Assessing the research landscape and clinical utility of large language models: a scoping review. BMC Med Inform Decis Mak. 2024 Mar 12;24(1):72. doi: 10.1186/s12911-024-02459-6. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-024-02459-6 .10.1186/s12911-024-02459-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Copilot. Microsoft. [2025-04-21]. https://www.microsoft.com/en-us/microsoft-copilot .
  • 8.Gemini. Google. [2025-04-21]. https://gemini.google.com/
  • 9.Llama. Meta. [2025-04-21]. https://llama.meta.com/
  • 10.Tsoutsanis P, Tsoutsanis A. Evaluation of large language model performance on the Multi-Specialty Recruitment Assessment (MSRA) exam. Comput Biol Med. 2024 Jan;168:107794. doi: 10.1016/j.compbiomed.2023.107794.S0010-4825(23)01259-3 [DOI] [PubMed] [Google Scholar]
  • 11.Shukla R, Mishra A, Banerjee N, Verma A. The comparison of ChatGPT 3.5, Microsoft Bing, and Google Gemini for diagnosing cases of neuro-ophthalmology. Cureus. 2024 Apr;16(4):e58232. doi: 10.7759/cureus.58232. https://europepmc.org/abstract/MED/38745784 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Pressman SM, Borna S, Gomez-Cabello CA, Haider SA, Forte AJ. AI in hand surgery: assessing large language models in the classification and management of hand injuries. J Clin Med. 2024 May 11;13(10):2832. doi: 10.3390/jcm13102832. https://www.mdpi.com/resolver?pii=jcm13102832 .jcm13102832 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Vaishya R, Iyengar KP, Patralekh MK, Botchu R, Shirodkar K, Jain VK, Vaish A, Scarlat MM. Effectiveness of AI-powered chatbots in responding to orthopaedic postgraduate exam questions-an observational study. Int Orthop. 2024 Aug 15;48(8):1963–1969. doi: 10.1007/s00264-024-06182-9.10.1007/s00264-024-06182-9 [DOI] [PubMed] [Google Scholar]
  • 14.Lee Y, Tessier L, Brar K, Malone S, Jin D, McKechnie T, Jung JJ, Kroh M, Dang JT, ASMBS Artificial Intelligence and Digital Surgery Taskforce Performance of artificial intelligence in bariatric surgery: comparative analysis of ChatGPT-4, Bing, and Bard in the American Society for Metabolic and Bariatric Surgery textbook of bariatric surgery questions. Surg Obes Relat Dis. 2024 Jul;20(7):609–613. doi: 10.1016/j.soard.2024.04.014. https://linkinghub.elsevier.com/retrieve/pii/S1550-7289(24)00169-2 .S1550-7289(24)00169-2 [DOI] [PubMed] [Google Scholar]
  • 15.Wei Q, Yao Z, Cui Y, Wei B, Jin Z, Xu X. Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis. J Biomed Inform. 2024 Mar;151:104620. doi: 10.1016/j.jbi.2024.104620. https://linkinghub.elsevier.com/retrieve/pii/S1532-0464(24)00038-8 .S1532-0464(24)00038-8 [DOI] [PubMed] [Google Scholar]
  • 16.Kaboudi N, Firouzbakht S, Shahir Eftekhar Mohammad, Fayazbakhsh Fatemeh, Joharivarnoosfaderani Niloufar, Ghaderi Salar, Dehdashti Mohammadreza, Mohtasham Kia Yasmin, Afshari Maryam, Vasaghi-Gharamaleki Maryam, Haghani Leila, Moradzadeh Zahra, Khalaj Fattaneh, Mohammadi Zahra, Hasanabadi Zahra, Shahidi Ramin. Diagnostic accuracy of ChatGPT for patients' triage; a systematic review and meta-analysis. Arch Acad Emerg Med. 2024;12(1):e60. doi: 10.22037/aaem.v12i1.2384. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Patil A, Serrato P, Chisvo N, Arnaout O, See PA, Huang KT. Large language models in neurosurgery: a systematic review and meta-analysis. Acta Neurochir (Wien) 2024 Nov 23;166(1):475. doi: 10.1007/s00701-024-06372-9.10.1007/s00701-024-06372-9 [DOI] [PubMed] [Google Scholar]
  • 18.Nguyen HC, Dang HP, Nguyen TL, Hoang V, Nguyen VA. Accuracy of latest large language models in answering multiple choice questions in dentistry: a comparative study. PLoS One. 2025 Jan 29;20(1):e0317423. doi: 10.1371/journal.pone.0317423. https://dx.plos.org/10.1371/journal.pone.0317423 .PONE-D-24-40356 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Lo CK, Mertz D, Loeb M. Newcastle-Ottawa Scale: comparing reviewers' to authors' assessments. BMC Med Res Methodol. 2014 Apr 01;14:45. doi: 10.1186/1471-2288-14-45. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-14-45 .1471-2288-14-45 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Long C, Subburam D, Lowe K, Dos Santos André, Zhang J, Hwang S, Saduka N, Horev Y, Su T, Côté David W J, Wright ED. ChatENT: augmented large language model for expert knowledge retrieval in otolaryngology-head and neck surgery. Otolaryngol Head Neck Surg. 2024 Oct 19;171(4):1042–1051. doi: 10.1002/ohn.864. [DOI] [PubMed] [Google Scholar]
  • 21.Tao BK, Hua N, Milkovich J, Micieli JA. ChatGPT-3.5 and Bing Chat in ophthalmology: an updated evaluation of performance, readability, and informative sources. Eye (Lond) 2024 Jul 20;38(10):1897–1902. doi: 10.1038/s41433-024-03037-w.10.1038/s41433-024-03037-w [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Shieh A, Tran B, He G, Kumar M, Freed JA, Majety P. Assessing ChatGPT 4.0's test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports. Sci Rep. 2024 Apr 23;14(1):9330. doi: 10.1038/s41598-024-58760-x. https://doi.org/10.1038/s41598-024-58760-x .10.1038/s41598-024-58760-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Sarangi PK, Narayan RK, Mohakud S, Vats A, Sahani D, Mondal H. Assessing the capability of ChatGPT, Google Bard, and Microsoft Bing in solving radiology case vignettes. Indian J Radiol Imaging. 2024 Apr 29;34(2):276–282. doi: 10.1055/s-0043-1777746. http://www.thieme-connect.com/DOI/DOI?10.1055/s-0043-1777746 .IJRI-23-9-2963 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Singer MB, Fu JJ, Chow J, Teng CC. Development and evaluation of Aeyeconsult: a novel ophthalmology chatbot leveraging verified textbook knowledge and GPT-4. J Surg Educ. 2024 Mar;81(3):438–443. doi: 10.1016/j.jsurg.2023.11.019.S1931-7204(23)00432-4 [DOI] [PubMed] [Google Scholar]
  • 25.Hanna RE, Smith LR, Mhaskar R, Hanna K. Performance of language models on the family medicine in-training exam. Fam Med. 2024 Oct 2;56(9):555–560. doi: 10.22454/fammed.2024.233738. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Kadoya N, Arai K, Tanaka S, Kimura Y, Tozuka R, Yasui K, Hayashi N, Katsuta Y, Takahashi H, Inoue K, Jingu K. Assessing knowledge about medical physics in language-generative AI with large language model: using the medical physicist exam. Radiol Phys Technol. 2024 Dec 10;17(4):929–937. doi: 10.1007/s12194-024-00838-2.10.1007/s12194-024-00838-2 [DOI] [PubMed] [Google Scholar]
  • 27.Sallam M, Al-Mahzoum K, Almutawaa RA, Alhashash JA, Dashti RA, AlSafy DR, Almutairi RA, Barakat M. The performance of OpenAI ChatGPT-4 and Google Gemini in virology multiple-choice questions: a comparative analysis of English and Arabic responses. BMC Res Notes. 2024 Sep 03;17(1):247. doi: 10.1186/s13104-024-06920-7. https://bmcresnotes.biomedcentral.com/articles/10.1186/s13104-024-06920-7 .10.1186/s13104-024-06920-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Gravina AG, Pellegrino R, Palladino G, Imperio G, Ventura A, Federico A. Charting new AI education in gastroenterology: cross-sectional evaluation of ChatGPT and perplexity AI in medical residency exam. Dig Liver Dis. 2024 Aug;56(8):1304–1311. doi: 10.1016/j.dld.2024.02.019.S1590-8658(24)00302-5 [DOI] [PubMed] [Google Scholar]
  • 29.Passby L, Jenko N, Wernham A. Performance of ChatGPT on specialty certificate examination in dermatology multiple-choice questions. Clin Exp Dermatol. 2024 Jun 25;49(7):722–727. doi: 10.1093/ced/llad197.7188526 [DOI] [PubMed] [Google Scholar]
  • 30.Sabri H, Saleh MHA, Hazrati P, Merchant K, Misch J, Kumar PS, Wang H, Barootchi S. Performance of three artificial intelligence (AI)-based large language models in standardized testing; implications for AI-assisted dental education. J Periodontal Res. 2025 Feb 18;60(2):121–133. doi: 10.1111/jre.13323. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Çamur Eren, Cesur T, Güneş Yasin Celal. Can large language models be new supportive tools in coronary computed tomography angiography reporting? Clin Imaging. 2024 Oct;114:110271. doi: 10.1016/j.clinimag.2024.110271.S0899-7071(24)00201-8 [DOI] [PubMed] [Google Scholar]
  • 32.Lubitz M, Latario L. Performance of two artificial intelligence generative language models on the orthopaedic in-training examination. Orthopedics. 2024 May;47(3):e146–e150. doi: 10.3928/01477447-20240304-02. [DOI] [PubMed] [Google Scholar]
  • 33.Gupta R, Hamid A, Jhaveri M, Patel N, Suthar P. Comparative evaluation of AI models such as ChatGPT 3.5, ChatGPT 4.0, and Google Gemini in neuroradiology diagnostics. Cureus. 2024 Aug;16(8):e67766. doi: 10.7759/cureus.67766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Lee G, Hong D, Kim S, Kim Jong Won, Lee Young Hwan, Park Sang O, Lee Kyeong Ryong. Comparison of the problem-solving performance of ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard for the Korean emergency medicine board examination question bank. Medicine (Baltimore) 2024 Mar 01;103(9):e37325. doi: 10.1097/MD.0000000000037325. https://europepmc.org/abstract/MED/38428889 .00005792-202403010-00048 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Is EE, Menekseoglu AK. Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o. Clin Rheumatol. 2024 Nov 28;43(11):3507–3513. doi: 10.1007/s10067-024-07154-5.10.1007/s10067-024-07154-5 [DOI] [PubMed] [Google Scholar]
  • 36.D'Anna Gennaro, Van Cauter S, Thurnher M, Van Goethem J, Haller S. Can large language models pass official high-grade exams of the European Society of Neuroradiology courses? A direct comparison between OpenAI chatGPT 3.5, OpenAI GPT4 and Google Bard. Neuroradiology. 2024 Aug 06;66(8):1245–1250. doi: 10.1007/s00234-024-03371-6.10.1007/s00234-024-03371-6 [DOI] [PubMed] [Google Scholar]
  • 37.Altamimi I, Alhumimidi A, Alshehri S, Alrumayan Abdullah, Al-Khlaiwi Thamir, Meo Sultan A, Temsah Mohamad-Hani. The scientific knowledge of three large language models in cardiology: multiple-choice questions examination-based performance. Ann Med Surg (Lond) 2024 Jun;86(6):3261–3266. doi: 10.1097/MS9.0000000000002120. https://europepmc.org/abstract/MED/38846858 .AMSU-D-23-02753 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Schoch J, Schmelz H, Strauch A, Borgmann H, Nestler T. Performance of ChatGPT-3.5 and ChatGPT-4 on the European Board of Urology (EBU) exams: a comparative analysis. World J Urol. 2024 Jul 26;42(1):445. doi: 10.1007/s00345-024-05137-4.10.1007/s00345-024-05137-4 [DOI] [PubMed] [Google Scholar]
  • 39.May M, Körner-Riffard Katharina, Kollitsch L, Burger M, Brookman-May SD, Rauchenwald M, Marszalek M, Eredics K. Evaluating the efficacy of AI chatbots as tutors in urology: a comparative analysis of responses to the 2022 In-Service Assessment of the European Board of Urology. Urol Int. 2024 Mar 30;108(4):359–366. doi: 10.1159/000537854. https://doi.org/10.1159/000537854 .000537854 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Sadeq MA, Ghorab RMF, Ashry MH, Abozaid AM, Banihani HA, Salem M, Aisheh MTA, Abuzahra S, Mourid MR, Assker MM, Ayyad M, Moawad MHED. AI chatbots show promise but limitations on UK medical exam questions: a comparative performance study. Sci Rep. 2024 Aug 14;14(1):18859. doi: 10.1038/s41598-024-68996-2. https://doi.org/10.1038/s41598-024-68996-2 .10.1038/s41598-024-68996-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Khalpey Z, Kumar U, King N, Abraham A, Khalpey A. Large language models take on cardiothoracic surgery: a comparative analysis of the performance of four models on American Board of Thoracic Surgery exam questions in 2023. Cureus. 2024 Jul;16(7):e65083. doi: 10.7759/cureus.65083. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Patel EA, Fleischer L, Filip P, Eggerstedt M, Hutz M, Michaelides E, Batra PS, Tajudeen BA. Comparative performance of ChatGPT 3.5 and GPT4 on rhinology standardized board examination questions. OTO Open. 2024 Jun 27;8(2):e164. doi: 10.1002/oto2.164. https://europepmc.org/abstract/MED/38938507 .OTO2164 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Irmici G, Cozzi A, Della Pepa G, De Berardinis C, D'Ascoli Elisa, Cellina M, Cè Maurizio, Depretto C, Scaperrotta G. How do large language models answer breast cancer quiz questions? A comparative study of GPT-3.5, GPT-4 and Google Gemini. Radiol Med. 2024 Oct 13;129(10):1463–1467. doi: 10.1007/s11547-024-01872-1.10.1007/s11547-024-01872-1 [DOI] [PubMed] [Google Scholar]
  • 44.Kollitsch L, Eredics K, Marszalek M, Rauchenwald M, Brookman-May SD, Burger M, Körner-Riffard Katharina, May M. How does artificial intelligence master urological board examinations? A comparative analysis of different large language models' accuracy and reliability in the 2022 In-Service Assessment of the European Board of Urology. World J Urol. 2024 Jan 10;42(1):20. doi: 10.1007/s00345-023-04749-6.10.1007/s00345-023-04749-6 [DOI] [PubMed] [Google Scholar]
  • 45.Morreel S, Verhoeven V, Mathysen D. Microsoft Bing outperforms five other generative artificial intelligence chatbots in the Antwerp University multiple choice medical license exam. PLOS Digit Health. 2024 Feb 14;3(2):e0000349. doi: 10.1371/journal.pdig.0000349. https://europepmc.org/abstract/MED/38354127 .PDIG-D-23-00311 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Bajčetić M, Mirčić A, Rakočević J, Đoković D, Milutinović K, Zaletel I. Comparing the performance of artificial intelligence learning models to medical students in solving histology and embryology multiple choice questions. Ann Anat. 2024 Jun;254:152261. doi: 10.1016/j.aanat.2024.152261.S0940-9602(24)00053-0 [DOI] [PubMed] [Google Scholar]
  • 47.Canillas Del Rey F, Canillas Arias M. Exploring the potential of artificial intelligence in traumatology: conversational answers to specific questions. Rev Esp Cir Ortop Traumatol. 2025 Jan;69(1):38–46. doi: 10.1016/j.recot.2024.05.004. https://linkinghub.elsevier.com/retrieve/pii/S1888-4415(24)00086-9 .S1888-4415(24)00086-9 [DOI] [PubMed] [Google Scholar]
  • 48.Meyer A, Riese J, Streichert T. Comparison of the performance of GPT-3.5 and GPT-4 with that of medical students on the written German Medical Licensing Examination: observational study. JMIR Med Educ. 2024 Feb 08;10:e50965. doi: 10.2196/50965. https://mededu.jmir.org/2024//e50965/ v10i1e50965 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Toyama Y, Harigai A, Abe M, Nagano M, Kawabata M, Seki Y, Takase K. Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society. Jpn J Radiol. 2024 Feb 04;42(2):201–207. doi: 10.1007/s11604-023-01491-2. https://europepmc.org/abstract/MED/37792149 .10.1007/s11604-023-01491-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Touma NJ, Caterini J, Liblk K. Is ChatGPT ready for primetime? Performance of artificial intelligence on a simulated Canadian urology board exam. Can Urol Assoc J. 2024 Oct 10;18(10):329–332. doi: 10.5489/cuaj.8800. https://doi.org/10.5489/cuaj.8800 .cuaj.8800 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Chan J, Dong T, Angelini G. The performance of large language models in intercollegiate Membership of the Royal College of Surgeons examination. Ann R Coll Surg Engl. 2024 Nov 06;106(8):700–704. doi: 10.1308/rcsann.2024.0023. https://publishing.rcseng.ac.uk/doi/10.1308/rcsann.2024.0023?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Patil NS, Huang RS, van der Pol CB, Larocque N. Comparative performance of ChatGPT and Bard in a text-based radiology knowledge assessment. Can Assoc Radiol J. 2024 May 14;75(2):344–350. doi: 10.1177/08465371231193716. https://journals.sagepub.com/doi/abs/10.1177/08465371231193716?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed . [DOI] [PubMed] [Google Scholar]
  • 53.Hubany S, Scala F, Hashemi K, Kapoor Saumya, Fedorova Julia R, Vaccaro Matthew J, Ridout Rees P, Hedman Casey C, Kellogg Brian C, Leto Barone Angelo A. ChatGPT-4 surpasses residents: a study of artificial intelligence competency in plastic surgery in-service examinations and its advancements from ChatGPT-3.5. Plast Reconstr Surg Glob Open. 2024 Sep;12(9):e6136. doi: 10.1097/GOX.0000000000006136.GOX-D-24-00262 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Nakajima N, Fujimori T, Furuya M, Kanie Yuya, Imai Hirotatsu, Kita Kosuke, Uemura Keisuke, Okada Seiji. A comparison between GPT-3.5, GPT-4, and GPT-4V: can the large language model (ChatGPT) pass the Japanese Board of Orthopaedic Surgery examination? Cureus. 2024 Mar;16(3):e56402. doi: 10.7759/cureus.56402. https://europepmc.org/abstract/MED/38633935 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Thibaut G, Dabbagh A, Liverneaux P. Does Google's Bard Chatbot perform better than ChatGPT on the European hand surgery exam? Int Orthop. 2024 Jan 15;48(1):151–158. doi: 10.1007/s00264-023-06034-y.10.1007/s00264-023-06034-y [DOI] [PubMed] [Google Scholar]
  • 56.Lum Z, Collins D, Dennison S, Guntupalli Lohitha, Choudhary Soham, Saiz Augustine M, Randall Robert L. Generative artificial intelligence performs at a second-year orthopedic resident level. Cureus. 2024 Mar;16(3):e56104. doi: 10.7759/cureus.56104. https://europepmc.org/abstract/MED/38618358 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57.Menekşeoğlu AK, İş EE. Comparative performance of artificial ıntelligence models in physical medicine and rehabilitation board-level questions. Rev Assoc Med Bras (1992) 2024;70(7):e20240241. doi: 10.1590/1806-9282.20240241. https://www.scielo.br/scielo.php?script=sci_arttext&pid=S0104-42302024000700614&lng=en&nrm=iso&tlng=en .S0104-42302024000700614 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Cheong RCT, Pang KP, Unadkat S, Mcneillis V, Williamson A, Joseph J, Randhawa P, Andrews P, Paleri V. Performance of artificial intelligence chatbots in sleep medicine certification board exams: ChatGPT versus Google Bard. Eur Arch Otorhinolaryngol. 2024 Apr;281(4):2137–2143. doi: 10.1007/s00405-023-08381-3.10.1007/s00405-023-08381-3 [DOI] [PubMed] [Google Scholar]
  • 59.Mesnard Benoît, Schirmann Aurélie, Branchereau Julien, Perrot Ophélie, Bogaert Guy, Neuzillet Yann, Lebret Thierry, Madec François-Xavier. Artificial intelligence: ready to pass the European Board examinations in urology? Eur Urol Open Sci. 2024 Feb;60:44–46. doi: 10.1016/j.euros.2024.01.002. https://linkinghub.elsevier.com/retrieve/pii/S2666-1683(24)00211-8 .S2666-1683(24)00211-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Ming S, Guo Q, Cheng W, Lei B. Influence of model evolution and system roles on ChatGPT's performance in Chinese medical licensing exams: comparative study. JMIR Med Educ. 2024 Aug 13;10:e52784–e52784. doi: 10.2196/52784. https://mededu.jmir.org/2024//e52784/ v10i1e52784 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Chow R, Hasan S, Zheng A, Gao C, Valdes G, Yu F, Chhabra A, Raman S, Choi JI, Lin H, Simone CB. The accuracy of artificial intelligence ChatGPT in oncology examination questions. J Am Coll Radiol. 2024 Nov;21(11):1800–1804. doi: 10.1016/j.jacr.2024.07.011. https://linkinghub.elsevier.com/retrieve/pii/S1546-1440(24)00675-6 .S1546-1440(24)00675-6 [DOI] [PubMed] [Google Scholar]
  • 62.Kim SE, Lee JH, Choi BS, Han H, Lee MC, Ro DH. Performance of ChatGPT on solving orthopedic board-style questions: a comparative analysis of ChatGPT 3.5 and ChatGPT 4. Clin Orthop Surg. 2024 Aug;16(4):669–673. doi: 10.4055/cios23179. https://www.ecios.org/DOIx.php?id=10.4055/cios23179 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Oura T, Tatekawa H, Horiuchi D, Matsushita S, Takita H, Atsukawa N, Mitsuyama Y, Yoshida A, Murai K, Tanaka R, Shimono T, Yamamoto A, Miki Y, Ueda D. Diagnostic accuracy of vision-language models on Japanese diagnostic radiology, nuclear medicine, and interventional radiology specialty board examinations. Jpn J Radiol. 2024 Dec 20;42(12):1392–1398. doi: 10.1007/s11604-024-01633-0.10.1007/s11604-024-01633-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Lewandowski M, Łukowicz Paweł, Świetlik Dariusz, Barańska-Rybak Wioletta. ChatGPT-3.5 and ChatGPT-4 dermatological knowledge level based on the Specialty Certificate Examination in Dermatology. Clin Exp Dermatol. 2024 Jun 25;49(7):686–691. doi: 10.1093/ced/llad255.7237242 [DOI] [PubMed] [Google Scholar]
  • 65.Knoedler L, Alfertshofer M, Knoedler S, Hoch CC, Funk PF, Cotofana S, Maheta B, Frank K, Brébant Vanessa, Prantl L, Lamby P. Pure wisdom or Potemkin villages? A comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE Step 3 style questions: quantitative analysis. JMIR Med Educ. 2024 Jan 05;10:e51148. doi: 10.2196/51148. https://mededu.jmir.org/2024//e51148/ v10i1e51148 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Khan AA, Yunus R, Sohail M, Rehman TA, Saeed S, Bu Y, Jackson CD, Sharkey A, Mahmood F, Matyal R. Artificial intelligence for anesthesiology board-style examination questions: role of large language models. J Cardiothorac Vasc Anesth. 2024 May;38(5):1251–1259. doi: 10.1053/j.jvca.2024.01.032.S1053-0770(24)00090-9 [DOI] [PubMed] [Google Scholar]
  • 67.Sheikh MS, Thongprayoon C, Qureshi F, Suppadungsuk S, Kashani KB, Miao J, Craici IM, Cheungpasitporn W. Personalized medicine transformed: ChatGPT's contribution to continuous renal replacement therapy alarm management in intensive care units. J Pers Med. 2024 Feb 22;14(3):233. doi: 10.3390/jpm14030233. https://www.mdpi.com/resolver?pii=jpm14030233 .jpm14030233 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Mayo-Yáñez Miguel, Lechien JR, Maria-Saibene A, Vaira LA, Maniaci A, Chiesa-Estomba CM. Examining the performance of ChatGPT 3.5 and Microsoft Copilot in otolaryngology: a comparative study with otolaryngologists' evaluation. Indian J Otolaryngol Head Neck Surg. 2024 Aug 01;76(4):3465–3469. doi: 10.1007/s12070-024-04729-1.4729 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69.Rydzewski NR, Dinakaran D, Zhao SG, Ruppin E, Turkbey B, Citrin DE, Patel KR. Comparative evaluation of LLMs in clinical oncology. NEJM AI. 2024 May 25;1(5):1. doi: 10.1056/aioa2300151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 70.Wang T, Mu J, Chen J, Lin C. Comparing ChatGPT and clinical nurses' performances on tracheostomy care: a cross-sectional study. Int J Nurs Stud Adv. 2024 Jun;6:100181. doi: 10.1016/j.ijnsa.2024.100181. https://linkinghub.elsevier.com/retrieve/pii/S2666-142X(24)00008-0 .S2666-142X(24)00008-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71.Liang R, Zhao A, Peng L, Xu X, Zhong J, Wu F, Yi F, Zhang S, Wu S, Hou J. Enhanced artificial intelligence strategies in renal oncology: iterative optimization and comparative analysis of GPT 3.5 versus 4.0. Ann Surg Oncol. 2024 Jun 12;31(6):3887–3893. doi: 10.1245/s10434-024-15107-0.10.1245/s10434-024-15107-0 [DOI] [PubMed] [Google Scholar]
  • 72.Jaworski A, Jasiński Dawid, Jaworski W, Hop Aleksandra, Janek Artur, Sławińska Barbara, Konieczniak Lena, Rzepka Maciej, Jung Maximilian, Sysło Oliwia, Jarząbek Victoria, Błecha Zuzanna, Haraziński Konrad, Jasińska Natalia. Comparison of the performance of artificial intelligence versus medical professionals in the Polish final medical examination. Cureus. 2024 Aug;16(8):e66011. doi: 10.7759/cureus.66011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Bharatha A, Ojeh N, Fazle Rabbi AM, Campbell M, Krishnamurthy K, Layne-Yarde R, Kumar A, Springer D, Connell K, Majumder MA. Comparing the performance of ChatGPT-4 and medical students on MCQs at varied levels of Bloom’s taxonomy. AMEP. 2024 May;Volume 15:393–400. doi: 10.2147/amep.s457408. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 74.Le M, Davis M. ChatGPT yields a passing score on a pediatric board preparatory exam but raises red flags. Global Pediatric Health. 2024 Mar 24;11:1. doi: 10.1177/2333794x241240327. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 75.Arango S, Flynn J, Zeitlin J, Lorenzana Daniel J, Miller Andrew J, Wilson Matthew S, Strohl Adam B, Weiss Lawrence E, Weir Tristan B. The performance of ChatGPT on the American Society for Surgery of the Hand self-assessment examination. Cureus. 2024 Apr;16(4):e58950. doi: 10.7759/cureus.58950. https://europepmc.org/abstract/MED/38800302 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 76.Rojas M, Rojas M, Burgess V, Toro-Pérez Javier, Salehi S. Exploring the performance of ChatGPT versions 3.5, 4, and 4 with vision in the Chilean medical licensing examination: observational study. JMIR Med Educ. 2024 Apr 29;10:e55048–e55048. doi: 10.2196/55048. https://mededu.jmir.org/2024//e55048/ v10i1e55048 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 77.Chau RCW, Thu KM, Yu OY, Hsung RT, Lo ECM, Lam WYH. Performance of generative artificial intelligence in dental licensing examinations. Int Dent J. 2024 Jun;74(3):616–621. doi: 10.1016/j.identj.2023.12.007. https://linkinghub.elsevier.com/retrieve/pii/S0020-6539(23)00989-9 .S0020-6539(23)00989-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 78.Thirunavukarasu AJ, Mahmood S, Malem A, Foster WP, Sanghera R, Hassan R, Zhou S, Wong SW, Wong YL, Chong YJ, Shakeel A, Chang Y, Tan BKJ, Jain N, Tan TF, Rauz S, Ting DSW, Ting DSJ. Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: a head-to-head cross-sectional study. PLOS Digit Health. 2024 Apr 17;3(4):e0000341. doi: 10.1371/journal.pdig.0000341. https://europepmc.org/abstract/MED/38630683 .PDIG-D-23-00293 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 79.Bicknell BT, Butler D, Whalen S, Ricks J, Dixon CJ, Clark AB, Spaedy O, Skelton A, Edupuganti N, Dzubinski L, Tate H, Dyess G, Lindeman B, Lehmann LS. ChatGPT-4 Omni performance in USMLE disciplines and clinical skills: comparative analysis. JMIR Med Educ. 2024 Nov 06;10:e63430. doi: 10.2196/63430. https://mededu.jmir.org/2024//e63430/ v10i1e63430 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 80.Haddad F, Saade JS. Performance of ChatGPT on ophthalmology-related questions across various examination levels: observational study. JMIR Med Educ. 2024 Jan 18;10:e50842. doi: 10.2196/50842. https://mededu.jmir.org/2024//e50842/ v10i1e50842 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 81.Noda R, Izaki Y, Kitano F, Komatsu J, Ichikawa D, Shibagaki Y. Performance of ChatGPT and Bard in self-assessment questions for nephrology board renewal. Clin Exp Nephrol. 2024 May 14;28(5):465–469. doi: 10.1007/s10157-023-02451-w.10.1007/s10157-023-02451-w [DOI] [PubMed] [Google Scholar]
  • 82.Yudovich MS, Makarova E, Hague CM, Raman JD. Performance of GPT-3.5 and GPT-4 on standardized urology knowledge assessment items in the United States: a descriptive study. J Educ Eval Health Prof. 2024 Jul 08;21:17. doi: 10.3352/jeehp.2024.21.17. https://dx.doi.org/10.3352/jeehp.2024.21.17 .jeehp.2024.21.17 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 83.Li D, Kao Y, Tsai S, Bai Y, Yeh T, Chu C, Hsu C, Cheng S, Hsu T, Liang C, Su K. Comparing the performance of ChatGPT GPT-4, Bard, and Llama-2 in the Taiwan Psychiatric Licensing Examination and in differential diagnosis with multi-center psychiatrists. Psychiatry Clin Neurosci. 2024 Jun 26;78(6):347–352. doi: 10.1111/pcn.13656. [DOI] [PubMed] [Google Scholar]
  • 84.Farhat F, Chaudhry BM, Nadeem M, Sohail SS, Madsen D? Evaluating large language models for the National Premedical Exam in India: comparative analysis of GPT-3.5, GPT-4, and Bard. JMIR Med Educ. 2024 Feb 21;10:e51523. doi: 10.2196/51523. https://mededu.jmir.org/2024//e51523/ v10i1e51523 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 85.Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023 Feb 08;9:e45312. doi: 10.2196/45312. https://mededu.jmir.org/2023//e45312/ v9i1e45312 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 86.Kung JE, Marshall C, Gauthier C, Gonzalez TA, Jackson JB. Evaluating ChatGPT performance on the orthopaedic in-training examination. JB JS Open Access. 2023;8(3):1. doi: 10.2106/JBJS.OA.23.00056. https://europepmc.org/abstract/MED/37693092 .JBJSOA-D-23-00056 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 87.Gencer A, Aydin S. Can ChatGPT pass the thoracic surgery exam? Am J Med Sci. 2023 Oct;366(4):291–295. doi: 10.1016/j.amjms.2023.08.001.S0002-9629(23)01292-2 [DOI] [PubMed] [Google Scholar]
  • 88.Ali R, Tang OY, Connolly ID, Zadnik Sullivan PL, Shin JH, Fridley JS, Asaad WF, Cielo D, Oyelese AA, Doberstein CE, Gokaslan ZL, Telfeian AE. Performance of ChatGPT and GPT-4 on neurosurgery written board examinations. Neurosurgery. 2023 Dec 01;93(6):1353–1365. doi: 10.1227/neu.0000000000002632.00006123-202312000-00018 [DOI] [PubMed] [Google Scholar]
  • 89.Massey PA, Montgomery C, Zhang AS. Comparison of ChatGPT-3.5, ChatGPT-4, and orthopaedic resident performance on orthopaedic assessment examinations. J Am Acad Orthop Surg. 2023 Dec 01;31(23):1173–1179. doi: 10.5435/JAAOS-D-23-00396. https://europepmc.org/abstract/MED/37671415 .00124635-990000000-00782 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 90.Suchman K, Garg S, Trindade AJ. Chat Generative Pretrained Transformer fails the multiple-choice American College of Gastroenterology self-assessment test. Am J Gastroenterol. 2023 Dec 01;118(12):2280–2282. doi: 10.14309/ajg.0000000000002320.00000434-202312000-00032 [DOI] [PubMed] [Google Scholar]
  • 91.Sakai D, Maeda T, Ozaki A, Kanda G, Kurimoto Y, Takahashi M. Performance of ChatGPT in board examinations for specialists in the Japanese Ophthalmology Society. Cureus. 2023 Dec;15(12):e49903. doi: 10.7759/cureus.49903. https://europepmc.org/abstract/MED/38174202 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 92.Huang Y, Gomaa A, Semrau S, Haderlein M, Lettmaier S, Weissmann T, Grigo J, Tkhayat HB, Frey B, Gaipl U, Distel L, Maier A, Fietkau R, Bert C, Putz F. Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for AI-assisted medical education and decision making in radiation oncology. Front Oncol. 2023;13:1265024. doi: 10.3389/fonc.2023.1265024. https://europepmc.org/abstract/MED/37790756 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 93.Yanagita Y, Yokokawa D, Uchida S, Tawara J, Ikusaka M. Accuracy of ChatGPT on medical questions in the National Medical Licensing Examination in Japan: evaluation study. JMIR Form Res. 2023 Oct 13;7:e48023. doi: 10.2196/48023. https://formative.jmir.org/2023//e48023/ v7i1e48023 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 94.Teebagy S, Colwell L, Wood E, Yaghy A, Faustina M. Improved performance of ChatGPT-4 on the OKAP Examination: a comparative study with ChatGPT-3.5. J Acad Ophthalmol (2017) 2023 Jul 11;15(2):e184–e187. doi: 10.1055/s-0043-1774399. http://www.thieme-connect.com/DOI/DOI?10.1055/s-0043-1774399 .JAO-425 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 95.Kaneda Y, Takahashi R, Kaneda U, Akashima S, Okita H, Misaki S, Yamashiro A, Ozaki A, Tanimoto T. Assessing the performance of GPT-3.5 and GPT-4 on the 2023 Japanese Nursing Examination. Cureus. 2023 Aug;15(8):e42924. doi: 10.7759/cureus.42924. https://europepmc.org/abstract/MED/37667724 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 96.Flores-Cohaila JA, García-Vicente Abigaíl, Vizcarra-Jiménez Sonia F, De la Cruz-Galán Janith P, Gutiérrez-Arratia Jesús D, Quiroga Torres BG, Taype-Rondan A. Performance of ChatGPT on the Peruvian National Licensing Medical Examination: cross-sectional study. JMIR Med Educ. 2023 Sep 28;9:e48039. doi: 10.2196/48039. https://mededu.jmir.org/2023//e48039/ v9i1e48039 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 97.Fowler T, Pullen S, Birkett L. Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions. Br J Ophthalmol. 2024 Sep 20;108(10):1379–1383. doi: 10.1136/bjo-2023-324091.bjo-2023-324091 [DOI] [PubMed] [Google Scholar]
  • 98.Moshirfar M, Altaf AW, Stoakes IM, Tuttle JJ, Hoopes PC. Artificial intelligence in ophthalmology: a comparative analysis of GPT-3.5, GPT-4, and human expertise in answering StatPearls questions. Cureus. 2023 Jun;15(6):e40822. doi: 10.7759/cureus.40822. https://europepmc.org/abstract/MED/37485215 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 99.Brin D, Sorin V, Vaid A, Soroush A, Glicksberg BS, Charney AW, Nadkarni G, Klang E. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci Rep. 2023 Oct 01;13(1):16492. doi: 10.1038/s41598-023-43436-9. https://doi.org/10.1038/s41598-023-43436-9 .10.1038/s41598-023-43436-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 100.Miao J, Thongprayoon C, Garcia Valencia OA, Krisanapan P, Sheikh MS, Davis PW, Mekraksakit P, Suarez MG, Craici IM, Cheungpasitporn W. Performance of ChatGPT on nephrology test questions. CJASN. 2023 Oct 18;19(1):35–43. doi: 10.2215/cjn.0000000000000330. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 101.Kaneda Y, Namba M, Kaneda U, Tanimoto T. Artificial intelligence in childcare: assessing the performance and acceptance of ChatGPT responses. Cureus. 2023 Aug;15(8):e44484. doi: 10.7759/cureus.44484. https://europepmc.org/abstract/MED/37791148 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 102.Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study. JMIR Med Educ. 2023 Jun 29;9:e48002. doi: 10.2196/48002. https://mededu.jmir.org/2023//e48002/ v9i1e48002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 103.Ali R, Tang OY, Connolly ID, Fridley JS, Shin JH, Zadnik Sullivan PL, Cielo D, Oyelese AA, Doberstein CE, Telfeian AE, Gokaslan ZL, Asaad WF. Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank. Neurosurgery. 2023 Nov 01;93(5):1090–1098. doi: 10.1227/neu.0000000000002551.00006123-990000000-00775 [DOI] [PubMed] [Google Scholar]
  • 104.Ohta K, Ohta S. The Performance of GPT-3.5, GPT-4, and Bard on the Japanese National Dentist Examination: a comparison study. Cureus. 2023 Dec;15(12):e50369. doi: 10.7759/cureus.50369. https://europepmc.org/abstract/MED/38213361 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 105.Watari T, Takagi S, Sakaguchi K, Nishizaki Y, Shimizu T, Yamamoto Y, Tokuda Y. Performance comparison of ChatGPT-4 and Japanese medical residents in the General Medicine In-Training Examination: comparison study. JMIR Med Educ. 2023 Dec 06;9:e52202. doi: 10.2196/52202. https://mededu.jmir.org/2023//e52202/ v9i1e52202 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 106.Roos J, Kasapovic A, Jansen T, Kaczmarczyk R. Artificial intelligence in medical education: comparative analysis of ChatGPT, Bing, and medical students in Germany. JMIR Med Educ. 2023 Sep 04;9:e46482. doi: 10.2196/46482. https://mededu.jmir.org/2023//e46482/ v9i1e46482 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 107.Guillen-Grima F, Guillen-Aguinaga S, Guillen-Aguinaga L, Alas-Brun R, Onambele L, Ortega W, Montejo R, Aguinaga-Ontoso E, Barach P, Aguinaga-Ontoso I. Evaluating the efficacy of ChatGPT in navigating the Spanish Medical Residency Entrance Examination (MIR): promising horizons for AI in clinical medicine. Clin Pract. 2023 Nov 20;13(6):1460–1487. doi: 10.3390/clinpract13060130. https://www.mdpi.com/resolver?pii=clinpract13060130 .clinpract13060130 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 108.Huang RS, Lu KJQ, Meaney C, Kemppainen J, Punnett A, Leung F. Assessment of resident and AI chatbot performance on the University of Toronto family medicine residency progress test: comparative study. JMIR Med Educ. 2023 Sep 19;9:e50514. doi: 10.2196/50514. https://mededu.jmir.org/2023//e50514/ v9i1e50514 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 109.Schubert MC, Wick W, Venkataramani V. Performance of large language models on a neurology board-style examination. JAMA Netw Open. 2023 Dec 01;6(12):e2346721. doi: 10.1001/jamanetworkopen.2023.46721. https://europepmc.org/abstract/MED/38060223 .2812620 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 110.Torres-Zegarra BC, Rios-Garcia W, Ñaña-Cordova Alvaro Micael, Arteaga-Cisneros KF, Chalco XCB, Ordoñez Marina Atena Bustamante, Rios CJG, Godoy CAR, Quezada KLTP, Gutierrez-Arratia JD, Flores-Cohaila JA. Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study. J Educ Eval Health Prof. 2023 Nov 20;20:30. doi: 10.3352/jeehp.2023.20.30. https://europepmc.org/abstract/MED/37981579 .jeehp.2023.20.30 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 111.Kirshteyn G, Golan R, Chaet M. Performance of ChatGPT vs. HuggingChat on OB-GYN topics. Cureus. 2024 Mar;16(3):e56187. doi: 10.7759/cureus.56187. https://europepmc.org/abstract/MED/38618446 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 112.van Nuland M, Erdogan A, Aςar Cenkay, Contrucci R, Hilbrants S, Maanach L, Egberts T, van der Linden PD. Performance of ChatGPT on factual knowledge questions regarding clinical pharmacy. J Clin Pharmacol. 2024 Sep 16;64(9):1095–1100. doi: 10.1002/jcph.2443. [DOI] [PubMed] [Google Scholar]
  • 113.Danesh A, Pazouki H, Danesh F, Danesh A, Vardar‐Sengul S. Artificial intelligence in dental education: ChatGPT's performance on the periodontic in‐service examination. Journal of Periodontology. 2024 Jan 10;95(7):682–687. doi: 10.1002/jper.23-0514. [DOI] [PubMed] [Google Scholar]
  • 114.Huang CY, Zhang E, Caussade M, Brown T, Stockton Hogrogian G, Yan AC. Pediatric dermatologists versus AI bots: evaluating the medical knowledge and diagnostic capabilities of ChatGPT. Pediatr Dermatol. 2024 May 09;41(5):831–834. doi: 10.1111/pde.15649. [DOI] [PubMed] [Google Scholar]
  • 115.Fiedler B, Azua EN, Phillips T, Ahmed AS. ChatGPT performance on the American Shoulder and Elbow Surgeons maintenance of certification exam. J Shoulder Elbow Surg. 2024 Sep;33(9):1888–1893. doi: 10.1016/j.jse.2024.02.029.S1058-2746(24)00231-3 [DOI] [PubMed] [Google Scholar]
  • 116.Coleman MC, Moore JN. Two artificial intelligence models underperform on examinations in a veterinary curriculum. J Am Vet Med Assoc. 2024 May 01;262(5):692–697. doi: 10.2460/javma.23.12.0666. https://avmajournals.avma.org/doi/10.2460/javma.23.12.0666 . [DOI] [PubMed] [Google Scholar]
  • 117.Abbas A, Rehman MS, Rehman SS. Comparing the performance of popular large language models on the National Board of Medical Examiners sample questions. Cureus. 2024 Mar;16(3):e55991. doi: 10.7759/cureus.55991. https://europepmc.org/abstract/MED/38606229 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 118.Jarou ZJ, Dakka A, McGuire D, Bunting L. ChatGPT versus human performance on emergency medicine board preparation questions. Ann Emerg Med. 2024 Jan;83(1):87–88. doi: 10.1016/j.annemergmed.2023.08.010.S0196-0644(23)00663-7 [DOI] [PubMed] [Google Scholar]
  • 119.Sensoy E, Citirik M. Assessing the proficiency of artificial intelligence programs in the diagnosis and treatment of cornea, conjunctiva, and eyelid diseases and exploring the advantages of each other benefits. Cont Lens Anterior Eye. 2024 Apr;47(2):102125. doi: 10.1016/j.clae.2024.102125.S1367-0484(24)00008-0 [DOI] [PubMed] [Google Scholar]
  • 120.Guerra GA, Hofmann HL, Le JL, Wong AM, Fathi A, Mayfield CK, Petrigliano FA, Liu JN. ChatGPT, Bard, and Bing chat are large language processing models that answered orthopaedic in-training examination questions with similar accuracy to first-year orthopaedic surgery residents. Arthroscopy. 2025 Mar;41(3):557–562. doi: 10.1016/j.arthro.2024.08.023.S0749-8063(24)00621-2 [DOI] [PubMed] [Google Scholar]
  • 121.Agarwal M, Goswami A, Sharma Priyanka. Evaluating ChatGPT-3.5 and Claude-2 in answering and explaining conceptual medical physiology multiple-choice questions. Cureus. 2023 Sep;15(9):e46222. doi: 10.7759/cureus.46222. https://europepmc.org/abstract/MED/37908959 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 122.Cheong KX, Zhang C, Tan T, Fenner BJ, Wong WM, Teo KY, Wang YX, Sivaprasad S, Keane PA, Lee CS, Lee AY, Cheung CMG, Wong TY, Cheong Y, Song SJ, Tham YC. Comparing generative and retrieval-based chatbots in answering patient questions regarding age-related macular degeneration and diabetic retinopathy. Br J Ophthalmol. 2024 Sep 20;108(10):1443–1449. doi: 10.1136/bjo-2023-324533.bjo-2023-324533 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 123.Zhou S, Luo X, Chen C, Jiang Hong, Yang Chun, Ran Guanghui, Yu Juan, Yin Chengliang. The performance of large language model-powered chatbots compared to oncology physicians on colorectal cancer queries. Int J Surg. 2024 Oct 01;110(10):6509–6517. doi: 10.1097/JS9.0000000000001850.01279778-990000000-01734 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 124.Kozaily E, Geagea M, Akdogan ER, Atkins J, Elshazly MB, Guglin M, Tedford RJ, Wehbe RM. Accuracy and consistency of online large language model-based artificial intelligence chat platforms in answering patients' questions about heart failure. Int J Cardiol. 2024 Aug 01;408:132115. doi: 10.1016/j.ijcard.2024.132115.S0167-5273(24)00737-X [DOI] [PubMed] [Google Scholar]
  • 125.Xia S, Hua Q, Mei Z, Xu W, Lai L, Wei M, Qin Y, Luo L, Wang C, Huo S, Fu L, Zhou F, Wu J, Zhang L, Lv D, Li J, Wang X, Li N, Song Y, Zhou J. Clinical application potential of large language model: a study based on thyroid nodules. Endocrine. 2025 Jan 30;87(1):206–213. doi: 10.1007/s12020-024-03981-3.10.1007/s12020-024-03981-3 [DOI] [PubMed] [Google Scholar]
  • 126.Lee Y, Shin T, Tessier L, Javidan A, Jung J, Hong D, Strong AT, McKechnie T, Malone S, Jin D, Kroh M, Dang JT, ASMBS Artificial Intelligence and Digital Surgery Task Force Harnessing artificial intelligence in bariatric surgery: comparative analysis of ChatGPT-4, Bing, and Bard in generating clinician-level bariatric surgery recommendations. Surg Obes Relat Dis. 2024 Jul;20(7):603–608. doi: 10.1016/j.soard.2024.03.011. https://linkinghub.elsevier.com/retrieve/pii/S1550-7289(24)00118-7 .S1550-7289(24)00118-7 [DOI] [PubMed] [Google Scholar]
  • 127.Doğan L, Özçakmakcı Gazi Bekir, Yılmaz E. The performance of chatbots and the AAPOS website as a tool for amblyopia education. J Pediatr Ophthalmol Strabismus. 2024 Apr 25;61(5):325–331. doi: 10.3928/01913913-20240409-01. [DOI] [PubMed] [Google Scholar]
  • 128.Lee T, Campbell D, Patel S, Hossain Afif, Radfar Navid, Siddiqui Emaad, Gardin Julius M. Unlocking health literacy: the ultimate guide to hypertension education from ChatGPT versus Google Gemini. Cureus. 2024 May;16(5):e59898. doi: 10.7759/cureus.59898. https://europepmc.org/abstract/MED/38721479 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 129.Lang SP, Yoseph ET, Gonzalez-Suarez AD, Kim R, Fatemi P, Wagner K, Maldaner N, Stienen MN, Zygourakis CC. Analyzing large language models' responses to common lumbar spine fusion surgery questions: a comparison between ChatGPT and Bard. Neurospine. 2024 Jun;21(2):633–641. doi: 10.14245/ns.2448098.049. https://europepmc.org/abstract/MED/38955533 .ns.2448098.049 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 130.Iannantuono G, Bracken-Clarke D, Karzai F, Choo-Wosoba H, Gulley J, Floudas C. Comparison of large language models in answering immuno-oncology questions: a cross-sectional study. Oncologist. 2024 May 03;29(5):407–414. doi: 10.1093/oncolo/oyae009. https://europepmc.org/abstract/MED/38309720 .7600405 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 131.Anguita R, Downie C, Ferro Desideri L, Sagoo MS. Assessing large language models' accuracy in providing patient support for choroidal melanoma. Eye (Lond) 2024 Nov 13;38(16):3113–3117. doi: 10.1038/s41433-024-03231-w.10.1038/s41433-024-03231-w [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 132.Zhang Y, Dong Y, Mei Z, Hou Y, Wei M, Yeung YH, Xu J, Hua Q, Lai L, Li N, Xia S, Zhou C, Zhou J. Performance of large language models on benign prostatic hyperplasia frequently asked questions. Prostate. 2024 Jun;84(9):807–813. doi: 10.1002/pros.24699. [DOI] [PubMed] [Google Scholar]
  • 133.Xue E, Bracken-Clarke D, Iannantuono GM, Choo-Wosoba H, Gulley JL, Floudas CS. Utility of large language models for health care professionals and patients in navigating hematopoietic stem cell transplantation: comparison of the performance of ChatGPT-3.5, ChatGPT-4, and Bard. J Med Internet Res. 2024 May 17;26:e54758. doi: 10.2196/54758. https://www.jmir.org/2024//e54758/ v26i1e54758 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 134.Cao JJ, Kwon DH, Ghaziani TT, Kwo P, Tse G, Kesselman A, Kamaya A, Tse JR. Large language models' responses to liver cancer surveillance, diagnosis, and management questions: accuracy, reliability, readability. Abdom Radiol (NY) 2024 Dec 01;49(12):4286–4294. doi: 10.1007/s00261-024-04501-7.10.1007/s00261-024-04501-7 [DOI] [PubMed] [Google Scholar]
  • 135.Monroe CL, Abdelhafez YG, Atsina K, Aman E, Nardo L, Madani MH. Evaluation of responses to cardiac imaging questions by the artificial intelligence large language model ChatGPT. Clin Imaging. 2024 Aug;112:110193. doi: 10.1016/j.clinimag.2024.110193. https://linkinghub.elsevier.com/retrieve/pii/S0899-7071(24)00123-2 .S0899-7071(24)00123-2 [DOI] [PubMed] [Google Scholar]
  • 136.Chervonski E, Harish KB, Rockman CB, Sadek M, Teter KA, Jacobowitz GR, Berland TL, Lohr J, Moore C, Maldonado TS. Generative artificial intelligence chatbots may provide appropriate informational responses to common vascular surgery questions by patients. Vascular. 2025 Feb 18;33(1):229–237. doi: 10.1177/17085381241240550. [DOI] [PubMed] [Google Scholar]
  • 137.Kassab J, Hadi El Hajjar A, Wardrop RM, Brateanu A. Accuracy of online artificial intelligence models in primary care settings. Am J Prev Med. 2024 Jun;66(6):1054–1059. doi: 10.1016/j.amepre.2024.02.006.S0749-3797(24)00060-6 [DOI] [PubMed] [Google Scholar]
  • 138.Al-Sharif E, Penteado R, Dib El Jalbout Nahia, Topilow Nicole J, Shoji Marissa K, Kikkawa Don O, Liu Catherine Y, Korn Bobby S. Evaluating the accuracy of ChatGPT and Google BARD in fielding oculoplastic patient queries: a comparative study on artificial versus human intelligence. Ophthalmic Plast Reconstr Surg. 2024;40(3):303–311. doi: 10.1097/IOP.0000000000002567.00002341-202405000-00010 [DOI] [PubMed] [Google Scholar]
  • 139.Mejia MR, Arroyave JS, Saturno M, Ndjonko LCM, Zaidat B, Rajjoub R, Ahmed W, Zapolsky I, Cho SK. Use of ChatGPT for determining clinical and surgical treatment of lumbar disc herniation with radiculopathy: a North American Spine Society guideline comparison. Neurospine. 2024 Mar;21(1):149–158. doi: 10.14245/ns.2347052.526. https://europepmc.org/abstract/MED/38291746 .ns.2347052.526 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 140.Lee T, Rao A, Campbell D, Radfar N, Dayal M, Khrais A. Evaluating ChatGPT-3.5 and ChatGPT-4.0 responses on hyperlipidemia for patient education. Cureus. 2024 May;16(5):e61067. doi: 10.7759/cureus.61067. https://europepmc.org/abstract/MED/38803402 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 141.Oliveira AL, Coelho M, Guedes LC, Cattoni MB, Carvalho H, Duarte-Batista P. Performance of ChatGPT 3.5 and 4 as a tool for patient support before and after DBS surgery for Parkinson's disease. Neurol Sci. 2024 Dec 29;45(12):5757–5764. doi: 10.1007/s10072-024-07732-0.10.1007/s10072-024-07732-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 142.Lim ZW, Pushpanathan K, Yew SME, Lai Y, Sun C, Lam JSH, Chen DZ, Goh JHL, Tan MCJ, Sheng B, Cheng C, Koh VTC, Tham Y. Benchmarking large language models' performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine. 2023 Sep;95:104770. doi: 10.1016/j.ebiom.2023.104770. https://linkinghub.elsevier.com/retrieve/pii/S2352-3964(23)00336-5 .S2352-3964(23)00336-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 143.Rahsepar AA, Tavakoli N, Kim GHJ, Hassani C, Abtin F, Bedayat A. How AI responds to common lung cancer questions: ChatGPT vs Google Bard. Radiology. 2023 Jun;307(5):e230922. doi: 10.1148/radiol.230922. [DOI] [PubMed] [Google Scholar]
  • 144.Pushpanathan K, Lim ZW, Er Yew SM, Chen DZ, Hui'En Lin HA, Lin Goh JH, Wong WM, Wang X, Jin Tan MC, Chang Koh VT, Tham Y. Popular large language model chatbots' accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries. iScience. 2023 Nov 17;26(11):108163. doi: 10.1016/j.isci.2023.108163. https://linkinghub.elsevier.com/retrieve/pii/S2589-0042(23)02240-X .S2589-0042(23)02240-X [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 145.Coskun BN, Yagiz B, Ocakoglu G, Dalkilic E, Pehlivan Y. Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatol Int. 2024 Mar 25;44(3):509–515. doi: 10.1007/s00296-023-05473-5.10.1007/s00296-023-05473-5 [DOI] [PubMed] [Google Scholar]
  • 146.King RC, Samaan JS, Yeo YH, Peng Y, Kunkel DC, Habib AA, Ghashghaei R. A multidisciplinary assessment of ChatGPT's knowledge of amyloidosis: observational study. JMIR Cardio. 2024 Apr 19;8:e53421. doi: 10.2196/53421. https://cardio.jmir.org/2024//e53421/ v8i1e53421 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 147.Pinto VBP, de Azevedo MF, Wroclawski ML, Gentile G, Jesus VLM, de Bessa Junior J, Nahas WC, Sacomani CAR, Sandhu JS, Gomes CM. Conformity of ChatGPT recommendations with the AUA/SUFU guideline on postprostatectomy urinary incontinence. Neurourol Urodyn. 2024 Apr 07;43(4):935–941. doi: 10.1002/nau.25442. [DOI] [PubMed] [Google Scholar]
  • 148.Momenaei B, Wakabayashi T, Shahlaee A, Durrani AF, Pandit SA, Wang K, Mansour HA, Abishek RM, Xu D, Sridhar J, Yonekawa Y, Kuriyan AE. Assessing ChatGPT-3.5 versus ChatGPT-4 performance in surgical treatment of retinal diseases: a comparative study. Ophthalmic Surg Lasers Imaging Retina. 2024 Aug;55(8):481–482. doi: 10.3928/23258160-20240227-02. https://journals.healio.com/doi/abs/10.3928/23258160-20240227-02?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed . [DOI] [PubMed] [Google Scholar]
  • 149.Stevenson E, Walsh C, Hibberd L. Can artificial intelligence replace biochemists? A study comparing interpretation of thyroid function test results by ChatGPT and Google Bard to practising biochemists. Ann Clin Biochem. 2024 Mar 20;61(2):143–149. doi: 10.1177/00045632231203473. [DOI] [PubMed] [Google Scholar]
  • 150.Dronkers EAC, Geneid A, Al Yaghchi C, Lechien JR. Evaluating the potential of AI chatbots in treatment decision-making for acquired bilateral vocal fold paralysis in adults. J Voice. 2024 Apr 06;:1. doi: 10.1016/j.jvoice.2024.02.020. https://linkinghub.elsevier.com/retrieve/pii/S0892-1997(24)00059-6 .S0892-1997(24)00059-6 [DOI] [PubMed] [Google Scholar]
  • 151.Rahimli Ocakoglu S, Coskun B. The emerging role of AI in patient education: a comparative analysis of LLM accuracy for pelvic organ prolapse. Med Princ Pract. 2024 Mar 25;33(4):330–7. doi: 10.1159/000538538. https://doi.org/10.1159/000538538 .000538538 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 152.Gandhi AP, Joesph FK, Rajagopal V, Aparnavi P, Katkuri S, Dayama S, Satapathy P, Khatib MN, Gaidhane S, Zahiruddin QS, Behera A. Performance of ChatGPT on the India undergraduate community medicine examination: cross-sectional study. JMIR Form Res. 2024 Mar 25;8:e49964. doi: 10.2196/49964. https://formative.jmir.org/2024//e49964/ v8i1e49964 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 153.Tariq R, Malik S, Khanna S. Evolving landscape of large language models: an evaluation of ChatGPT and Bard in answering patient queries on colonoscopy. Gastroenterology. 2024 Jan;166(1):220–221. doi: 10.1053/j.gastro.2023.08.033.S0016-5085(23)04916-8 [DOI] [PubMed] [Google Scholar]
  • 154.Li P, Zhang X, Zhu E, Yu S, Sheng B, Tham YC, Wong TY, Ji H. Potential multidisciplinary use of large language models for addressing queries in cardio-oncology. J Am Heart Assoc. 2024 Mar 19;13(6):e033584. doi: 10.1161/JAHA.123.033584. https://www.ahajournals.org/doi/abs/10.1161/JAHA.123.033584?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 155.Sosa BR, Cung M, Suhardi VJ, Morse K, Thomson A, Yang HS, Iyer S, Greenblatt MB. Capacity for large language model chatbots to aid in orthopedic management, research, and patient queries. J Orthop Res. 2024 Jun 21;42(6):1276–1282. doi: 10.1002/jor.25782. [DOI] [PubMed] [Google Scholar]
  • 156.Koga S, Martin NB, Dickson DW. Evaluating the performance of large language models: ChatGPT and Google Bard in generating differential diagnoses in clinicopathological conferences of neurodegenerative disorders. Brain Pathol. 2024 May 08;34(3):e13207. doi: 10.1111/bpa.13207. https://europepmc.org/abstract/MED/37553205 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 157.Warrier A, Singh R, Haleem A, Zaki H, Eloy JA. The comparative diagnostic capability of large language models in otolaryngology. Laryngoscope. 2024 Sep 02;134(9):3997–4002. doi: 10.1002/lary.31434. [DOI] [PubMed] [Google Scholar]
  • 158.Kumar RP, Sivan V, Bachir H, Sarwar SA, Ruzicka F, O'Malley GR, Lobo P, Morales IC, Cassimatis ND, Hundal JS, Patel NV. Can artificial intelligence mitigate missed diagnoses by generating differential diagnoses for neurosurgeons? World Neurosurg. 2024 Jul;187:e1083–e1088. doi: 10.1016/j.wneu.2024.05.052.S1878-8750(24)00814-3 [DOI] [PubMed] [Google Scholar]
  • 159.Hirosawa T, Harada Y, Mizuta K, Sakamoto T, Tokumasu K, Shimizu T. Diagnostic performance of generative artificial intelligences for a series of complex case reports. Digit Health. 2024 Jul 21;10:20552076241265215. doi: 10.1177/20552076241265215. https://journals.sagepub.com/doi/10.1177/20552076241265215?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed .10.1177_20552076241265215 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 160.Mandalos A, Tsouris D. Artificial versus human intelligence in the diagnostic approach of ophthalmic case scenarios: a qualitative evaluation of performance and consistency. Cureus. 2024 Jun;16(6):e62471. doi: 10.7759/cureus.62471. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 161.Krusche M, Callhoff J, Knitza J, Ruffer N. Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatol Int. 2024 Feb 24;44(2):303–306. doi: 10.1007/s00296-023-05464-6. https://europepmc.org/abstract/MED/37742280 .10.1007/s00296-023-05464-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 162.Delsoz M, Madadi Y, Raja H, Munir WM, Tamm B, Mehravaran S, Soleimani M, Djalilian A, Yousefi S. Performance of ChatGPT in diagnosis of corneal eye diseases. Cornea. 2024 May 01;43(5):664–670. doi: 10.1097/ICO.0000000000003492.00003226-202405000-00019 [DOI] [PubMed] [Google Scholar]
  • 163.Kozel G, Gurses ME, Gecici NN, Gökalp Elif, Bahadir S, Merenzon MA, Shah AH, Komotar RJ, Ivan ME. Chat-GPT on brain tumors: an examination of artificial intelligence/machine learning's ability to provide diagnoses and treatment plans for example neuro-oncology cases. Clin Neurol Neurosurg. 2024 Apr;239:108238. doi: 10.1016/j.clineuro.2024.108238.S0303-8467(24)00125-2 [DOI] [PubMed] [Google Scholar]
  • 164.Stoneham S, Livesey A, Cooper H, Mitchell C. ChatGPT versus clinician: challenging the diagnostic capabilities of artificial intelligence in dermatology. Clin Exp Dermatol. 2024 Jun 25;49(7):707–710. doi: 10.1093/ced/llad402.7429032 [DOI] [PubMed] [Google Scholar]
  • 165.Albaladejo A, Lorleac'h A, Allain J. [The spring of artificial intelligence: AI vs. expert for internal medicine cases] Rev Med Interne. 2024 Jul;45(7):409–414. doi: 10.1016/j.revmed.2024.01.012. https://linkinghub.elsevier.com/retrieve/pii/S0248-8663(24)00032-8 .S0248-8663(24)00032-8 [DOI] [PubMed] [Google Scholar]
  • 166.Zandi R, Fahey JD, Drakopoulos M, Bryan JM, Dong S, Bryar PJ, Bidwell AE, Bowen RC, Lavine JA, Mirza RG. Exploring diagnostic precision and triage proficiency: a comparative study of GPT-4 and Bard in addressing common ophthalmic complaints. Bioengineering (Basel) 2024 Jan 26;11(2):120. doi: 10.3390/bioengineering11020120. https://www.mdpi.com/resolver?pii=bioengineering11020120 .bioengineering11020120 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 167.Hirosawa T, Kawamura R, Harada Y, Mizuta K, Tokumasu K, Kaji Y, Suzuki T, Shimizu T. ChatGPT-generated differential diagnosis lists for complex case-derived clinical vignettes: diagnostic accuracy evaluation. JMIR Med Inform. 2023 Oct 09;11:e48808. doi: 10.2196/48808. https://medinform.jmir.org/2023//e48808/ v11i1e48808 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 168.Hirosawa T, Harada Y, Yokose M, Sakamoto T, Kawamura R, Shimizu T. Diagnostic accuracy of differential-diagnosis lists generated by Generative Pretrained Transformer 3 Chatbot for clinical vignettes with common chief complaints: a pilot study. Int J Environ Res Public Health. 2023 Feb 15;20(4):3378. doi: 10.3390/ijerph20043378. https://www.mdpi.com/resolver?pii=ijerph20043378 .ijerph20043378 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 169.Fraser H, Crossland D, Bacher I, Ranney M, Madsen T, Hilliard R. Comparison of diagnostic and triage accuracy of Ada Health and WebMD symptom checkers, ChatGPT, and Physicians for Patients in an emergency department: clinical data analysis study. JMIR Mhealth Uhealth. 2023 Oct 03;11:e49995. doi: 10.2196/49995. https://mhealth.jmir.org/2023//e49995/ v11i1e49995 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 170.Rojas-Carabali W, Cifuentes-González Carlos, Wei X, Putera I, Sen A, Thng ZX, Agrawal R, Elze T, Sobrin L, Kempen JH, Lee B, Biswas J, Nguyen QD, Gupta V, de-la-Torre A, Agrawal R. Evaluating the diagnostic accuracy and management recommendations of ChatGPT in uveitis. Ocul Immunol Inflamm. 2024 Oct 18;32(8):1526–1531. doi: 10.1080/09273948.2023.2253471. [DOI] [PubMed] [Google Scholar]
  • 171.Gräf Markus, Knitza J, Leipe J, Krusche M, Welcker M, Kuhn S, Mucke J, Hueber AJ, Hornig J, Klemm P, Kleinert S, Aries P, Vuillerme N, Simon D, Kleyer A, Schett G, Callhoff J. Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy. Rheumatol Int. 2022 Dec;42(12):2167–2176. doi: 10.1007/s00296-022-05202-4. https://europepmc.org/abstract/MED/36087130 .10.1007/s00296-022-05202-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 172.Ward M, Unadkat P, Toscano D, Kashanian Alon, Lynch Daniel G, Horn Alexander C, D'Amico Randy S, Mittler Mark, Baum Griffin R. A quantitative assessment of ChatGPT as a neurosurgical triaging tool. Neurosurgery. 2024 Aug 01;95(2):487–495. doi: 10.1227/neu.0000000000002867.00006123-990000000-01055 [DOI] [PubMed] [Google Scholar]
  • 173.Hirosawa T, Mizuta K, Harada Y, Shimizu T. Comparative evaluation of diagnostic accuracy between Google Bard and physicians. Am J Med. 2023 Nov;136(11):1119–1123.e18. doi: 10.1016/j.amjmed.2023.08.003.S0002-9343(23)00536-3 [DOI] [PubMed] [Google Scholar]
  • 174.Lyons RJ, Arepalli SR, Fromal O, Choi JD, Jain N. Artificial intelligence chatbot performance in triage of ophthalmic conditions. Can J Ophthalmol. 2024 Aug;59(4):e301–e308. doi: 10.1016/j.jcjo.2023.07.016.S0008-4182(23)00234-X [DOI] [PubMed] [Google Scholar]
  • 175.Makhoul M, Melkane AE, Khoury PE, Hadi CE, Matar N. A cross-sectional comparative study: ChatGPT 3.5 versus diverse levels of medical experts in the diagnosis of ENT diseases. Eur Arch Otorhinolaryngol. 2024 May 16;281(5):2717–2721. doi: 10.1007/s00405-024-08509-z.10.1007/s00405-024-08509-z [DOI] [PubMed] [Google Scholar]
  • 176.Shemer A, Cohen M, Altarescu A, Atar-Vardi M, Hecht I, Dubinsky-Pertzov B, Shoshany N, Zmujack S, Or L, Einan-Lifshitz A, Pras E. Diagnostic capabilities of ChatGPT in ophthalmology. Graefes Arch Clin Exp Ophthalmol. 2024 Jul 06;262(7):2345–2352. doi: 10.1007/s00417-023-06363-z.10.1007/s00417-023-06363-z [DOI] [PubMed] [Google Scholar]
  • 177.Gunes Y, Cesur T. The diagnostic performance of large language models and general radiologists in thoracic radiology cases: a comparative study. J Thorac Imaging. 2024 Sep 13;:2024. doi: 10.1097/RTI.0000000000000805.00005382-990000000-00153 [DOI] [PubMed] [Google Scholar]
  • 178.Sarangi PK, Irodi A, Panda S, Nayak DSK, Mondal H. Radiological differential diagnoses based on cardiovascular and thoracic imaging patterns: perspectives of four large language models. Indian J Radiol Imaging. 2024 Apr 28;34(2):269–275. doi: 10.1055/s-0043-1777289. http://www.thieme-connect.com/DOI/DOI?10.1055/s-0043-1777289 .IJRI-23-9-2923 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 179.Berg HT, van Bakel B, van de Wouw L, Jie KE, Schipper A, Jansen H, O'Connor Rory D, van Ginneken B, Kurstjens S. ChatGPT and generating a differential diagnosis early in an emergency department presentation. Ann Emerg Med. 2024 Jan;83(1):83–86. doi: 10.1016/j.annemergmed.2023.08.003.S0196-0644(23)00642-X [DOI] [PubMed] [Google Scholar]
  • 180.Haider SA, Pressman SM, Borna S, Gomez-Cabello CA, Sehgal A, Leibovich BC, Forte AJ. Evaluating large language model (LLM) performance on established breast classification systems. Diagnostics (Basel) 2024 Jul 11;14(14):1491. doi: 10.3390/diagnostics14141491. https://www.mdpi.com/resolver?pii=diagnostics14141491 .diagnostics14141491 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 181.Gan RK, Ogbodo JC, Wee YZ, Gan AZ, González Pedro Arcos. Performance of Google bard and ChatGPT in mass casualty incidents triage. Am J Emerg Med. 2024 Jan;75:72–78. doi: 10.1016/j.ajem.2023.10.034. https://linkinghub.elsevier.com/retrieve/pii/S0735-6757(23)00576-4 .S0735-6757(23)00576-4 [DOI] [PubMed] [Google Scholar]
  • 182.Aiumtrakul N, Thongprayoon C, Arayangkool C, Vo KB, Wannaphut C, Suppadungsuk S, Krisanapan P, Garcia Valencia OA, Qureshi F, Miao J, Cheungpasitporn W. Personalized medicine in urolithiasis: AI chatbot-assisted dietary management of oxalate for kidney stone prevention. J Pers Med. 2024 Jan 18;14(1):107. doi: 10.3390/jpm14010107. https://www.mdpi.com/resolver?pii=jpm14010107 .jpm14010107 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 183.Wang H, Gao C, Dantona C, Hull B, Sun J. DRG-LLaMA: tuning LLaMA model to predict diagnosis-related group for hospitalized patients. NPJ Digit Med. 2024 Jan 22;7(1):16. doi: 10.1038/s41746-023-00989-3. https://doi.org/10.1038/s41746-023-00989-3 .10.1038/s41746-023-00989-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 184.Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole-Lewis H, Pfohl S, Payne P, Seneviratne M, Gamble P, Kelly C, Babiker A, Schärli Nathanael, Chowdhery A, Mansfield P, Demner-Fushman D, Agüera Y Arcas Blaise, Webster D, Corrado GS, Matias Y, Chou K, Gottweis J, Tomasev N, Liu Y, Rajkomar A, Barral J, Semturs C, Karthikesalingam A, Natarajan V. Large language models encode clinical knowledge. Nature. 2023 Aug;620(7972):172–180. doi: 10.1038/s41586-023-06291-2. https://europepmc.org/abstract/MED/37438534 .10.1038/s41586-023-06291-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 185.Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023 Aug 17;29(8):1930–1940. doi: 10.1038/s41591-023-02448-8.10.1038/s41591-023-02448-8 [DOI] [PubMed] [Google Scholar]
  • 186.Researcher Access Program application. OpenAI. [2025-04-21]. https://platform.openai.com/docs/model-index-for-researchers .
  • 187.Zhou J, He X, Sun L, Xu J, Chen X, Chu Y, Zhou L, Liao X, Zhang B, Afvari S, Gao X. Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4. Nat Commun. 2024 Jul 05;15(1):5649. doi: 10.1038/s41467-024-50043-3. https://doi.org/10.1038/s41467-024-50043-3 .10.1038/s41467-024-50043-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 188.Li Y, Li Z, Zhang K, Dan R, Jiang S, Zhang Y. ChatDoctor: a medical chat model fine-tuned on a large language model meta-AI (LLaMA) using medical domain knowledge. Cureus. 2023 Jun;15(6):e40895. doi: 10.7759/cureus.40895. https://europepmc.org/abstract/MED/37492832 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 189.Bagde H, Dhopte A, Alam MK, Basri R. A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research. Heliyon. 2023 Dec;9(12):e23050. doi: 10.1016/j.heliyon.2023.e23050. https://linkinghub.elsevier.com/retrieve/pii/S2405-8440(23)10258-1 .S2405-8440(23)10258-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 190.Levin G, Horesh N, Brezinov Y, Meyer R. Performance of ChatGPT in medical examinations: a systematic review and a meta-analysis. BJOG. 2024 Feb;131(3):378–380. doi: 10.1111/1471-0528.17641. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia Appendix 1

PRISMA checklist.

jmir_v27i1e64486_app1.docx (28.4KB, docx)
Multimedia Appendix 2

Search strategy.

jmir_v27i1e64486_app2.docx (13.9KB, docx)
Multimedia Appendix 3

Versions and timelines of LLMs iterations.

jmir_v27i1e64486_app3.docx (20.3KB, docx)
Multimedia Appendix 4

Examples of the outcomes.

jmir_v27i1e64486_app4.docx (1.3MB, docx)
Multimedia Appendix 5

Description of 168 studies included.

jmir_v27i1e64486_app5.docx (195.9KB, docx)
Multimedia Appendix 6

Quality assessment of observational study.

jmir_v27i1e64486_app6.docx (66KB, docx)
Multimedia Appendix 7

Indirect comparison results.

jmir_v27i1e64486_app7.docx (1.3MB, docx)
Multimedia Appendix 8

Node splitting inconsistency test.

jmir_v27i1e64486_app8.docx (2.5MB, docx)
Multimedia Appendix 9

Trace and density plots.

jmir_v27i1e64486_app9.docx (3.5MB, docx)
Multimedia Appendix 10

Objective questions are stratified according to different fields of the questions.

jmir_v27i1e64486_app10.docx (13KB, docx)

Data Availability Statement

The data sets generated or analyzed during this study are available from the corresponding author on reasonable request.

Articles from Journal of Medical Internet Research are provided here courtesy of JMIR Publications Inc.

RESOURCES