Skip to main content
Medical Science Educator logoLink to Medical Science Educator
. 2024 Nov 13;35(1):555–567. doi: 10.1007/s40670-024-02206-6

ChatGPT and Other Large Language Models in Medical Education — Scoping Literature Review

Alexandra Aster 1,✉,#, Matthias Carl Laupichler 1,#, Tamina Rockwell-Kollmann 1, Gilda Masala 1, Ebru Bala 1, Tobias Raupach 1
PMCID: PMC11933646  PMID: 40144083

Abstract

This review aims to provide a summary of all scientific publications on the use of large language models (LLMs) in medical education over the first year of their availability. A scoping literature review was conducted in accordance with the PRISMA recommendations for scoping reviews. Five scientific literature databases were searched using predefined search terms. The search yielded 1509 initial results, of which 145 studies were ultimately included. Most studies assessed LLMs’ capabilities in passing medical exams. Some studies discussed advantages, disadvantages, and potential use cases of LLMs. Very few studies conducted empirical research. Many published studies lack methodological rigor. We therefore propose a research agenda to improve the quality of studies on LLM.

Keywords: ChatGPT, Large language model, Artificial intelligence, Generative AI, Medical education

Introduction

The Rise of Large Language Models

For several years now [1, 2], high-performance artificial intelligence applications capable of understanding and generating natural language [3] have been existing, which are collectively referred to as “large language models” (LLMs). However, the general population became aware of this development mostly after November 30, 2022, when the freely accessible LLM “Chat Generative Pre-trained Transformer” (abbreviated as ChatGPT) was released by OpenAI [4]. ChatGPT amazes many users because it generates responses that are often indistinguishable from those of humans. While many other LLMs from various providers were released shortly after the launch of ChatGPT [5], particularly in popular media, ChatGPT is most popular in many parts of the world.

ChatGPT and other LLMs have already had a significant impact on various research fields in the first months since their release. Examples can be found in the most diverse fields, including software engineering [6], journalism [7], and finance [8]. Two areas stand out particularly, with practitioners in both domains expecting a great benefit from the use of LLMs: healthcare [9] and education [10]. In the healthcare sector, researchers currently anticipate promising application opportunities across three key areas [9, 11]: medical research (e.g., scientific text production and science communication), patient care/clinical practice (e.g., patient empowerment and improved documentation), and medical education. Educators, on the other hand, anticipate a positive impact of LLMs on learning processes (e.g., providing real-time feedback and improving students’ writing styles) as well as on teaching processes (e.g., planning lessons and enhancing assessment) [10]. Besides the manifold possibilities of LLMs, representatives from both disciplines also point out potential risks associated with their use. For example, Omiye et al. [12] found that many LLMs propagate “harmful, inaccurate, race-based” medicine [12]. In the field of education, other potential practical and ethical limitations have been discussed, such as the lack of replicability of results [13].

Apparently, the use of LLMs is being discussed intensely in both the medical and education domains, raising questions about the influence of LLMs on the intersection of the two fields: medical education.

LLMs in Medical Education

An example that has garnered significant attention in the medical community in recent months, fueling considerations regarding the use of LLMs in medical education, was ChatGPT’s ability to pass high-stakes medical examinations. Kung et al. [14] demonstrated relatively early on that ChatGPT can pass the United States Medical Licensing Examination (USMLE). Results like these have led other researchers to assert that these findings shine “a spotlight on the flaws of medical education” [15] as it may not require a human brain and mind to pass these exams. However, beyond the remarkable exam performance of ChatGPT, medical educators also seem to have recognized that AI-assisted LLMs like ChatGPT can have a significant positive as well as negative impact on the training of future physicians [16]. Yet, attempting to gain an overview of published research findings on this topic leads to a research landscape characterized by uninformed statements and questionable methods.

Aims and Scope

While a multitude of review papers on the use of AI in medical education already exist at the time of writing [17, 18], this review aims to provide a brief summary of the insights gained from 1 year of using LLMs such as ChatGPT in medical education. Given that the availability of ChatGPT and other chatbot LLMs offers a low-threshold entry point for the use of AI in medical education, we anticipate that AI-supported educational applications will play an increasingly significant role in medical schools.

Thus, this review aims to provide a comprehensive overview of the state of the literature on the use of LLMs in medical education, precisely 1 year after the publication of ChatGPT. Our main objective was to investigate the level of scientific interest (measured by the number of scientific publications) in LLMs in medical education shortly after the publication of the first freely accessible LLMs. We also wanted to examine the rough thematic foci and publication outlets (e.g., journals, conferences) of the articles as well as the global distribution of the associated research papers. Therefore, our first research question was:

RQ1: What is the extent of publications regarding LLMs in medical education 1 year after the release of the most prominent LLM, ChatGPT?

The next logical step is the examination of themes addressed in publications on LLMs in medical education. As mentioned earlier, some researchers appear to focus on ChatGPT’s capabilities in answering medical exams. Others, however, utilize LLMs to generate new exam questions [19]. Additionally, there is a plethora of opinion pieces balancing the benefits of implementing LLMs in medical schools against potential limitations [20]. An overview of the addressed themes would be helpful for medical educators, as it would enable them to easily identify what has already been done and which areas for future research appear most promising. Therefore, research question 2 was as follows:

RQ2: What specific themes do publications exploring the utilization of LLMs in medical education focus on?

Given the short time frame between the advent of LLMs (i.e., November 2022) and publication dates (up until November 2023), one could critically question the empirical quality of articles that plan, conduct, evaluate, and publish a complete study within but a few months. It is therefore reasonable to assume that a large portion of the publications is not based on empirical findings but rather on conceptual and theoretical considerations. While conceptual articles certainly have their value, it is still important to examine the ratio of the different types of publications. Hence, research question 3 read as follows:

RQ3: What is the relation of empirical research to theoretical/conceptual publications regarding LLMs in medical education?

Finally, it would also be helpful to know which medical specialties are taking on pioneering roles in the field of LLM in medical education research. Since LLMs belong to the family of natural language processing AI applications, it could be assumed that conversation-heavy disciplines such as psychotherapy, psychiatry, and to some extent even family medicine have a particularly keen interest in the use of LLMs in their teaching.

RQ4: Which medical specialties are currently involved in the exploration of LLMs within medical education?

Method

In conducting this scoping literature review, we adhered to the recommendations outlined in the “Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR)” by Tricco et al. [21].

Eligibility Criteria

We applied a total of six eligibility criteria for selecting appropriate publications (see Table 1). These criteria can be understood as hierarchically organized rules, whereby we first assessed coverage of the first eligibility criterion for each report. If this criterion was met, the second criterion was examined, and so forth. Only reports meeting all six eligibility criteria were included in the review.

Table 1.

Eligibility criteria

No Eligibility criteria Exclusion criteria
1 Published after Jan 1, 20001 Published before Jan 1, 20001
2 Abstract and main text were published in English or German Abstract and main text were published in another language
3 Published in a scientific journal or as a conference paper at a scientific conference or as part of a scientific book/collection Published in other outlets such as newspapers, magazines, websites, social media, and non-scientific books
4 Focused on the application or influence of ChatGPT or another LLM (Bard, Bing Chat, etc.) Report is not focused on LLMs
5 Focused solely on academic and vocational healthcare profession education (medical education, nursing education, continuing medical education, etc.) Focused on education in other disciplines (engineering education, legal education, etc.)
6 Focused on education (in healthcare professions) Not focused on education (e.g., using ChatGPT for clinical practice or medical research)

The same eligibility criteria were employed in both steps of the review process (i.e., abstract and full-text assessment, see section “Search Strategy”). Eligibility criterion 3 refers solely to the type of scientific outlet the report was published in, not on the article type (research articles, opinion pieces, etc.)

1While AI-supported natural language processing, and particularly large language models (LLMs), has only gained relevance in the past few years, we chose the year 2000 as the starting point to ensure that we did not overlook any potential pioneering articles. Additionally, there were instances where the databases (e.g., PubMed) listed incorrect publication dates.

Search Strategy

We searched the following scientific literature databases: ACM Digital Library (Association for Computing Machinery), IEEE Xplore (IEEE), JSTOR (ITHAKA), PubMed (National Library of Medicine), and Web of Science (Clarivate). We did not search other databases and did not use search engines to find gray literature. However, due to the novelty of the topic, we did include preprints that have not yet been peer-reviewed, as long as they were listed in one of the databases mentioned above. We conducted the search on November 30, 2023, exactly 1 year after the publication of OpenAI’s ChatGPT platform. Fig. 1 provides a more detailed breakdown of the number of records retrieved from the respective databases. We used the following combination of search terms and boolean operators in each of the five databases: (“LLM” OR “LLMs” OR “large language model*” OR “GPT” OR “ChatGPT” OR “chatbot*”) AND (“medical education” OR “medical school*” OR “medical exam*” OR “medical assessment*” OR “medical curricul*” OR “healthcare education” OR “continuing medical education” OR “internship” OR “residen*”).

Fig. 1.

Fig. 1

Flow diagram of the literature review process. The figure, which was adapted from Page et al. [22], was slightly modified, since we relied exclusively on databases and registers for study identification

Review Strategy

We employed a two-step literature review strategy. In step 1, we used Rayyan (rayyan.ai) [23], a software designed to help with organizing and conducting literature reviews. We retrieved the search results from all databases and registers, encompassing titles, abstracts, and bibliographic details, and subsequently imported the generated files into Rayyan. Following this, two authors (AA and GM) conducted an independent evaluation of each report based on the predefined exclusion criteria (see section “Eligibility Criteria”). During the evaluation stage, each rater was blinded to the decision of the other rater. Once all abstracts had been assessed, the blind-review mode was lifted, and any disparities were addressed and resolved through systematic discussion. In step 2, we sought to retrieve all reports whose titles and abstracts indicated that they were eligible for inclusion. Two authors (AA and TRK) independently read the remaining reports in their entirety and checked them against the exclusion criteria. The decisions were subsequently compared, and discrepancies were discussed under the moderation of another author (MCL). Inter-rater reliability, expressed as Cohen’s kappa, was ϰ = 0.911, which can be interpreted as nearly perfect agreement [24]. We calculated Cohen’s ϰ using the online calculator provided by Hemmerich [25]. Finally, a content analysis table was developed to extract the most relevant information from the studies, which were needed to answer the four research questions. The content analysis table consisted of several questions (organized in columns), each referring to a specific characteristic of the respective paper (organized in rows). For instance, the table included questions about the journal in which the paper was published or the thematic focus on which the paper concentrated. Due to the large number of included studies, the completion of the table was divided between authors, with the first and second halves of the entries being completed by two authors (AA and TRK for the first half and MCL and EB for the second half, respectively). Thus, the content of each study was independently analyzed by two authors. The content analysis procedure was conducted in Microsoft Excel (Microsoft, Version 2016), and data visualization was done via RStudio (Posit Software, Version 2023).

Results

Selection of Sources of Evidence and RQ 1: Number of Publications

Across all databases, a total of 1509 records were obtained of which 305 (20%) were excluded due to being duplicates. The titles and abstracts of 1204 (80%) records were subsequently checked for eligibility. After the exclusion of 1021 (68%) records, the full texts of 181 (12%) records were sought for retrieval. Since 24 (2%) records were not retrievable, the remaining 158 (10%) reports were assessed for eligibility. The most frequent reason for exclusion was reason 6 (see Table 1) with eight reports that had to be excluded, followed by reason 4 (n = 4) and reason 2 (n = 1). Consequently, 145 articles (10%) were included in the review (see Fig. 1) [14, 15, 20, 26–166].

The distribution of the publications encompassed a total of 80 journals and two preprint servers. The largest number of studies was published in JMIR Medical Education (n = 19), directly followed by Cureus (n = 18). Few studies were published in leading medical education journals such as Academic Medicine (n = 5), Medical Education (n = 1), or Medical Teacher (n = 1). In the majority of journals, only one publication was found. Seven reports were found on preprint servers (arXiv.org and medrXiv.org).

The corresponding authors of the publications came from 29 different countries (see Fig. 2). The countries with the most publications were the USA (n = 52, 36%) and India (n = 19, 13%).

Fig. 2.

Fig. 2

Map of the corresponding authors’ origin. We found no publications from authors of countries highlighted in gray. A deeper shade of blue indicates more publications on LLMs in medical education; a lighter shade of blue indicates less (but not zero) publications

RQ 2: Thematic Foci

In order to gain an overview of the thematic foci of the studies, we assigned each study to an overarching focus (see Table 2 presenting an overview of the foci including the references). The majority of studies (n = 75, 52%) focused on demonstrating ChatGPT’s capabilities in passing medical exams (e.g., USMLE, NMLE of different countries) or answering board exam preparation questions (e.g. neurosurgery oral board preparation question bank [70]). Other studies assessed ChatGPT’s performance in medical school-specific exams (e.g., dermatology [59], orthopedics [152], ophthalmology [87]). Across all exams, ChatGPT-4 seems to perform better than ChatGPT-3.5. For more information on the general distribution across medical specialties, please refer to Table 3.

Table 2.

Overview of the thematic foci

Foci Reference
Empirical Non-empirical

Empirical research based on collected data (qualitative or quantitative) without research on ChatGPT’s capabilities in passing medical exams

(n = 24)

[32–34, 41, 51, 57, 76, 77, 56, 94, 96, 110, 115, 117, 119, 142, 145, 146, 151, 153, 154, 156, 161, 164]
Demonstrating ChatGPT’s capabilities in passing medical exams (n = 75) [14, 26, 27, 31, 49, 50, 55, 60–66, 68–74, 80, 82, 84, 86–89, 91, 95, 97–109, 111–114, 116, 120, 122–124, 127–129, 131, 133, 135–139, 141, 143, 144, 147–150, 152, 155, 157, 159, 160, 165]

Conceptual papers developing possible scenario cases for LLMs in medical education

(n = 37)

[15, 28, 30, 35–37, 39, 40, 42–48, 52, 53, 58, 75, 78, 79, 81, 83, 85, 90, 92, 93, 118, 121, 125, 130, 132, 134, 140, 158, 162, 163]
Others (n = 9) [20, 29, 38, 54, 56, 59, 67, 126, 166]

Table 3.

Number of publications across various medical specialties

Medical specialty Absolute number Relative number
Several/not specified 67 46%
Ophthalmology, including neuro-ophthalmology 10 7%
General surgery 6 4%
Physiology 5 3%
Neurosurgery; emergency medicine (including advanced cardiac life support, medical intensive care, and resuscitation life support) 4 each 3%
Anesthesiology; biochemistry; plastic surgery; otolaryngology 3 each 2%
Anatomy; dermatology; general medicine; internal medicine and surgery; medical microbiology; neurology and neuroscience; nursing; obstetrics and gynecology; orthopedics; orthopedic surgery 2 each 1%
Bioethics; biomedicine; biostatistics; cardiology; clinical informatics; clinical toxicology; gastroenterology; hematology; molecular medicine; parasitology; pediatric genitourinary; pharmacology; public health; radiology; radiation oncology; thoracic surgery; urology 1 each 1%

Medical specialties are based on the term used by the respective authors. Different medical specialties are separated by semicolons

The second most common category under which studies could be subsumed was conceptual papers (n = 37, 26%) that focus on possible scenarios in which LLMs can be used in medical education. These publications discussed opportunities, advantages, disadvantages, and possibilities for the application of LLMs in different medical fields or more generally how LLMs can be used in medical education. Most authors reflect or discuss potentials simultaneously with pitfalls [46–48, 78, 79, 93, 121]. One example of a potential pitfall mentioned by the authors of the studies is the possibility of using ChatGPT to cheat [90, 92]. The advantages mentioned by the authors include, for example, the possibility of automatically creating exams or assigning grades and feedback [75, 90]. Regarding the use for medical students, ChatGPT seemed to function as a learning resource providing a personalized learning experience, or reference work as well as a helpful tool for summarizing studies [83, 132]. In the field of medical education, the authors proposed that ChatGPT may lend itself for simulating patients adequately enough to run a comprehensive anamnesis [118]. Moreover, ChatGPT could be used to assist the writing process of recommendation letters in terms of avoiding implicit biases [140], rating recommendation letters [30], or generating students’ personal statements [28]. Other authors presented conversations in the form of a dialogue conducted with ChatGPT [36, 39, 125, 130]. These conceptual papers are mostly published in the form of editorials, letters to the editor, etc. For more information on the ratio between empirical and conceptual research, refer to RQ 3.

Another category of studies (n = 24, 17%) focused on empirical research based on collected data in either a qualitative or quantitative manner. The aims of the empirical studies were manifold: Some authors used ChatGPT to generate or rate already generated multiple-choice questions [34, 94, 161, 164], or compared the performance of different LLMs on written surgical case scenarios [56]. Other authors compared ChatGPT’s and students’ performance in a parasitology examination [156], or used ChatGPT as a tool for students to check their own responses against ChatGPT’s responses on complex pediatric case vignettes [76]. Another study comparing characteristics of ChatGPT’s responses to surgical problems with short summaries from textbooks found that students rated evidence-based textbook summaries better in regard to comprehensiveness, but ChatGPT’s summaries better in regard to clarity and organization [33]. Another study focused on the empirical comparison of ChatGPT’s performance on state examination questions from different countries [51]. The authors found significant differences with ChatGPT performing best for the Italian and worst for the French examination [51]. Further studies used ChatGPT for writing personal statements [153] and comparing it to statements written by humans [41]. Additionally, a study assessed the general usage of ChatGPT and the attitudes of faculty member towards it in a medical school in Antigua [32] while other studies developed surveys to assess the use of LLMs in the context of medical education [145, 151].

Lastly, n = 9 (6%) studies focused on other topics that were not covered by the three previous categories. This category includes systematic or scoping reviews, responses to commentaries, and other articles.

RQ 3: Ratio Between Empirical and Conceptual Research

Regarding the included empirical research, the majority of studies (n = 86, 59%) took a quantitative approach mostly focusing on ChatGPT’s ability to pass medical exams (n = 68, 79%). Besides, seven studies represented qualitative empirical research (5%) and four studies were literature reviews (3%). The second largest amount of studies (n = 48, 33%) were narrative articles such as conceptual or opinion pieces (letter to the editor, editorials, etc.). Therefore, the ratio between conceptual and empirical research is 1.8:1.

RQ4: Medical Specialties

As can be seen in Table 3, a variety of medical specialties was currently engaged in using LLMs during the first year after the introduction of ChatGPT. All studies that specified the medical specialty concentrated solely only on the reported specialty while no study covered multiple specialties.

Discussion

General Discussion

This scoping review aimed at assessing the current status of the use of LLMs in medical education 1 year after the launch of the best known LLM, ChatGPT. To do so, we pursued the areas of four research questions: the extent of publications, thematic foci, the ratio between empirical research, and conceptual publications, as well as the spectrum of medical specialties. In line with the preceding literature regarding the use of LLMs in healthcare [9] as well as in education [11], our results emphasized that LLMs are used in medical education at larger scale. Specifically, the 145 included studies mostly focused on ChatGPT’s capabilities in passing medical exams, followed by conceptual research concentrating on advantages, disadvantages, and possible application scenarios of LLMs in medical education. Besides the assumption that healthcare and education anticipate the use of LLMs in a practical manner, our results showed that the trend of the included studies was on conceptual ideas of possible applications for LLMs. Only 24 of the included studies used an experimental study design (either qualitative or quantitative). The utilization of LLMs was restricted to a particular medical specialty but extended across various specialties from more theoretical ones (e.g., biostatistics) to more practical ones (e.g., thoracic surgery). These results showed that LLMs are not primarily used in conversation-heavy disciplines such as psychotherapy and psychiatry, but rather in data-heavy and practical disciplines.

Limitations

Several limitations may affect the validity of the findings. Firstly, a significant portion of the included studies was categorized as quantitative empirical research, even though no empirical study in the sense of a randomized controlled trial was conducted. This is due to the fact that most studies thematically focused on ChatGPT’s capability to pass medical exams. While these studies indeed presented quantitative data, they often did not follow a traditional study design (some intervention/manipulation, comparison with a control group, etc.). While the ability of LLMs to pass medical exams is certainly impressive, the contribution of these studies to medical education research and practice is questionable.

Another point to consider is the heterogeneous landscape of studies being published rapidly across various journals. In the span of 1 year, more than 145 studies on the topic were published in 80 different journals. Only five included studies were published in reputable journals like Academic Medicine, while no study was published in Medical Teacher, Medical Education, or Advances in Health Sciences Education. Although the focus of this review was on the use of LLMs in medical education, studies were frequently published in specialty-specific journals (e.g., American Journal of Gastroenterology, British Journal of Anaesthesia) having a high impact in the respective specialty. It would therefore be reasonable to wonder whether the associated rapid peer review and publication process could have impaired the methodological rigor and quality of the research. This assumption is also supported by the fact that few studies employed a rigorous study design, but primarily captured ChatGPT’s (mostly superficial) performance in medical exams.

However, in conducting our literature review, we also had to make some decisions that, although crucial to the feasibility of the study, could potentially limit the validity of the results. First, we refrained from explicitly searching for gray literature as we set the focus to studies published in peer-reviewed journals to ensure the inclusion of reliable high-quality primary studies. Nevertheless, we included some preprints assuming that these studies (though maybe with changes) will be published in the near future. Furthermore, in our search terms, we did not include other typical commercial LLMs such as Google’s “Bard” (later “Gemini”) or Microsoft’s “Copilot,” in addition to OpenAI’s ChatGPT. This decision was based on several factors, the primary one being that the majority of studies either utilized ChatGPT directly or referred to the generic term “large language model”. Since several articles were not retrievable, this may have affected the results of this review. Nonetheless, the sheer number of studies included suggests that the status of the current literature landscape is adequately depicted. Two independent reviewers rated each article, though not consistently by the same pair due to the number of included studies. While this might have affected the consistency of ratings, the interrater reliabilities showed satisfactory agreement values, suggesting that choosing this method was a reasonable decision.

Implications for Future Research and Practice

Conclusively, we propose some implications for future research from our findings. It became evident that most studies assessed LLM’s (particularly ChatGPT’s) performance in medical exams. We identified the research gap that studies are mostly lacking a theoretical valid research question as well as a sound empirical approach. While it is important to understand LLM’s medical proficiency level, drawing definite conclusions becomes challenging when studies lack an empirical approach and merely engage a LLM in question-answering tasks. As a consequence of the research gap identified, strengthening the empirical approach seems to be the most important implication for future research. Particular attention should be paid to developing a theory-driven research question which contains a sound-dependent variable that is assessed by suitable measurement methods. Measurement methods as well as the evaluation of the study should be aligned to Kirkpatrick’s four-level evaluation model to assure the effectiveness of the study design. A randomized controlled pretest–posttest (true experimental) [167] that compares an experimental group to a control group should be capable of providing reliable outcomes and analyzing causal relationships. Accordingly, a group of medical students learning with a LLM could be compared with a control group of medical students learning with traditional methods (e.g., textbook case vignettes).

LLMs used in educational settings, such as medical education, should meet several requirements. First and foremost, LLMs should be able to provide their users with correct information that should be as up-to-date as possible. In the field of medical education, it is essential that LLMs do not only provide correct information but also supplement it with medical specialized information. Therefore, future studies should also look at the development of medicine-specific LLMs, such as MedPaLM2. Another possibility to harness the benefits of LLMs for medical education might be using customizable LLM-interfaces, such as ChatGPT’s new “GPT”-creation feature, in which users (e.g., medical educators) can customize their own versions. In this context, it is thinkable of GPTs customized exclusively for the use in medical education.

Regarding the support of medical teachers, LLMs could be used to generate exam questions whose quality should be constantly rated by experts in the field of medical education as well as by medical students or residents (17). It is also thinkable that LLMs can be further used to provide medical students with feedback on essays or free-text responses. Thus, it must be investigated whether the feedback was correct and valuable, and whether it has an impact on students’ learning outcomes. Besides the support of medical teachers or students, LLMs could also be used to enhance practical skills, exemplarily by providing prompts that describe a step-by-step-instruction for specific procedures, such as CPR. Additionally, other generative AI applications that are able to generate images and videos instead of text may provide further opportunities for future research. It is conceivable that generative AI may be used for medical imaging specialties education, exemplarily the generation of X-ray images for pneumothorax.

Hence, future studies should address the overarching question of how LLMs can and should be employed in medical education. Should LLMs be used to relieve or support medical teachers or can students rely on LLMs to accompany or even enhance their learning process?

Conclusion

This scoping review shed a light on the literature landscape dealing with the use of LLMs in medical education. Our results showed that although a large number of studies were published within a year since the implementation of ChatGPT, only few studies applied a rigorous study design. Thus, we call on medical education and LLM researchers to conduct more studies following a stringent and methodologically sound research design in order to answer meaningful research questions.

Acknowledgements

None.

Abbreviations

AI

Artificial intelligence

ChatGPT

Chat Generative Pre-trained Transformer

LLM

Large language model

Author Contribution

AA and MCL were instrumental in conceptualizing the study, interpreting the data, and drafting the initial manuscript. GM and MCL were responsible for data collection. All authors participated in the data analysis. GM, TRK, and EB were involved in revising the manuscript. All authors have reviewed the final manuscript and consented to its submission to Medical Science Educator.

Funding

Open Access funding enabled and organized by Projekt DEAL.

Data Availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethical Approval

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Competing Interests

The authors declare no competing interests.

Footnotes

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Alexandra Aster and Matthias Carl Laupichler contributed equally to the article.

References

  • 1.Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In: Guyon I, Von Luxburg U, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Advances in Neural Information Processing Systems 30 (NIPS 2017). Long Beach, CA; 2017.
  • 2.Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. preprint. 2018.
  • 3.Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, et al. ChatGPT and other large language models are double-edged swords. Radiology. 2023;307(2):e230163. [DOI] [PubMed] [Google Scholar]
  • 4.OpenAI. https://openai.com/blog/chatgpt/. 2022. ChatGPT: optimizing language models for dialogue.
  • 5.Naveed H, Khan AU, Qiu S, Saqib M, Anwar S, Usman M, et al. A comprehensive overview of large language models. preprint. 2023;
  • 6.Hou X, Zhao Y, Liu Y, Yang Z, Wang K, Li L, et al. Large language models for software engineering: a systematic literature review. preprint. 2023.
  • 7.Petridis S, Diakopoulos N, Crowston K, Hansen M, Henderson K, Jastrzebski S, et al. AngleKindling: supporting journalistic angle ideation with large language models. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM; 2023. p. 1–16.
  • 8.Wu S, Irsoy O, Lu S, Dabravolski V, Dredze M, Gehrmann S, et al. BloombergGPT: a large language model for finance. preprint. 2023;
  • 9.Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930–40. [DOI] [PubMed] [Google Scholar]
  • 10.Kasneci E, Sessler K, Küchemann S, Bannert M, Dementieva D, Fischer F, et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learn Individ Differ [Internet]. 2023;103:102274. Available from: https://www.sciencedirect.com/science/article/pii/S1041608023000195
  • 11.Clusmann J, Kolbinger FR, Muti HS, Carrero ZI, Eckardt JN, Laleh NG, et al. The future landscape of large language models in medicine. Commun Med. 2023;3(1):141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Omiye JA, Lester JC, Spichak S, Rotemberg V, Daneshjou R. Large language models propagate race-based medicine. NPJ Digit Med. 2023;6(1):195. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Yan L, Sha L, Zhao L, Li Y, Martinez-Maldonado R, Chen G, et al. Practical and ethical challenges of large language models in education: a systematic scoping review. Br J Edu Technol. 2024;55(1):90–112. [Google Scholar]
  • 14.Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023;2(2): e0000198. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Mbakwe AB, Lourentzou I, Celi LA, Mechanic OJ, Dagan A. ChatGPT passing USMLE shines a spotlight on the flaws of medical education. PLOS Digital Health. 2023;2(2): e0000205. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Sallam M. The utility of ChatGPT as an example of large language models in healthcare education, research and practice: systematic review on the future perspectives and potential limitations. preprint. 2023
  • 17.Chan KS, Zary N. Applications and challenges of implementing artificial intelligence in medical education: integrative review. JMIR Med Educ. 2019;5(1): e13930. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Zhang W, Cai M, Lee HJ, Evans R, Zhu C, Ming C. AI in medical education: global situation, effects and challenges. Educ Inf Technol. 2024;29(4):4611–33. [Google Scholar]
  • 19.Laupichler MC, Rother JF, Grunwald Kadow IC, Ahmadi S, Raupach T. Large language models in medical education: comparing ChatGPT- to human-generated exam questions. Academic Medicine [Internet]. 9900; Available from: https://journals.lww.com/academicmedicine/fulltext/9900/large_language_models_in_medical_education_.719.aspx [DOI] [PubMed]
  • 20.Preiksaitis C, Rose C. Opportunities, challenges, and future directions of generative artificial intelligence in medical education: scoping review. JMIR Med Educ. 2023;20(9): e48785. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73. [DOI] [PubMed] [Google Scholar]
  • 22.Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88: 105906. [DOI] [PubMed] [Google Scholar]
  • 23.Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977;33(2):363. [PubMed] [Google Scholar]
  • 25.Hemmerich W. https://statistikguru.de/rechner/cohens-kappa-zwei-rater-berechnen.html. 2019. StatistikGuru: Cohen’s Kappa für zwei Rater berechnen.
  • 26.Smith J, Choi PM, Buntine P. Will code one day run a code? Performance of language models on <scp>ACEM</scp> primary examinations and implications. Emerg Med Australas. 2023;35(5):876–8. [DOI] [PubMed] [Google Scholar]
  • 27.Kufel J, Paszkiewicz I, Bielówka M, Bartnikowska W, Janik M, Stencel M, et al. Will ChatGPT pass the Polish specialty exam in radiology and diagnostic imaging? Insights into strengths and limitations. Pol J Radiol. 2023;88(1):430–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Zumsteg JM, Junn C. Will ChatGPT match to your program? Am J Phys Med Rehabil. 2023;102(6):545–7. [DOI] [PubMed] [Google Scholar]
  • 29.Epstein RH, Dexter F. Variability in large language models’ responses to medical licensing and certification examinations. Comment on “How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment.” JMIR Med Educ. 2023;9:e48305. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Barrett A, Hekman L, Ellis JL, Baldea KG, Bresler L. Utilization of ChatGPT for appraising letters of recommendation in urology residency applications: ready for prime time? J Urol. 2023;210(6):833–4. [DOI] [PubMed] [Google Scholar]
  • 31.Kao YS, Chuang WK, Yang J. Use of ChatGPT on Taiwan’s examination for medical doctors. Ann Biomed Eng. 2024;52(3):455–7. [DOI] [PubMed] [Google Scholar]
  • 32.Cross J, Robinson R, Devaraju S, Vaughans A, Hood R, Kayalackakom T, et al. Transforming medical education: assessing the integration of ChatGPT into faculty workflows at a Caribbean medical school. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 33.Breeding T, Martinez B, Patel H, Nasef H, Arif H, Nakayama D, et al. The utilization of ChatGPT in reshaping future medical education and learning perspectives: a curse or a blessing? Am Surg. 2024;90(4):560–6. [DOI] [PubMed] [Google Scholar]
  • 34.Ilgaz HB, Çelik Z. The significance of artificial intelligence platforms in anatomy education: an experience with ChatGPT and Google Bard. Cureus. 2023 Sep 15; [DOI] [PMC free article] [PubMed]
  • 35.Safranek CW, Sidamon-Eristoff AE, Gilson A, Chartash D. The role of large language models in medical education: applications and implications. JMIR Med Educ. 2023;14(9): e50945. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Eysenbach G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers. JMIR Med Educ. 2023;6(9): e46885. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Lee H. The rise of <scp>ChatGPT</scp> : exploring its potential in medical education. Anat Sci Educ. 2023. [DOI] [PubMed]
  • 38.Mohammad B, Supti T, Alzubaidi M, Shah H, Alam T, Shah Z, et al. The pros and cons of using ChatGPT in medical education: a scoping review. In 2023. [DOI] [PubMed]
  • 39.Totlis T, Natsis K, Filos D, Ediaroglou V, Mantzou N, Duparc F, et al. The potential role of ChatGPT and artificial intelligence in anatomy education: a conversation with ChatGPT. Surg Radiol Anat. 2023;45(10):1321–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Koga S. The potential of ChatGPT in medical education: focusing on USMLE preparation. Ann Biomed Eng. 2023;51(10):2123–4. [DOI] [PubMed] [Google Scholar]
  • 41.Patel V, Deleonibus A, Wells MW, Bernard SL, Schwarz GS. Distinguishing authentic voices in the age of ChatGPT: comparing AI-generated and applicant-written personal statements for plastic surgery residency application. Ann Plast Surg. 2023;91(3):324–5. [DOI] [PubMed] [Google Scholar]
  • 42.Wang LKP, Paidisetty PS, Cano AM. The next paradigm shift? ChatGPT, artificial intelligence, and medical education. Med Teach. 2023;45(8):925–925. [DOI] [PubMed] [Google Scholar]
  • 43.Ahn S. The impending impacts of large language models on medical education. Korean J Med Educ. 2023;35(1):103–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Heng JJY, Teo DB, Tan LF. The impact of Chat Generative Pre-trained Transformer (ChatGPT) on medical education. Postgrad Med J. 2023;99(1176):1125–7. [DOI] [PubMed] [Google Scholar]
  • 45.Liaw W, Chavez S, Pham C, Tehami S, Govender R. The hazards of using ChatGPT: a call to action for medical education researchers. PRiMER. 2023;7:27. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Bin Arif T, Munaf U, Ul-Haque I. The future of medical education and research: is ChatGPT a blessing or blight in disguise? Med Educ Online. 2023;28(1):2181052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Liu J, Liu F, Fang J, Liu S. The application of Chat Generative Pre-trained Transformer in nursing education. Nurs Outlook. 2023;71(6): 102064. [DOI] [PubMed] [Google Scholar]
  • 48.Karabacak M, Ozkara BB, Margetis K, Wintermark M, Bisdas S. The advent of generative language models in medical education. JMIR Med Educ. 2023;6(9): e48163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Elango A, Kannan N, Anandan I, Surapaneni K. Testing the knowledge and interpretation skills of ChatGPT in pharmacology examination of phase II MBBS. Indian J Pharmacol. 2023;55(4):266. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Panthier C, Gatinel D. Success of ChatGPT, an AI language model, in taking the French language version of the European Board of Ophthalmology examination: a novel approach to medical knowledge assessment. J Fr Ophtalmol. 2023;46(7):706–11. [DOI] [PubMed] [Google Scholar]
  • 51.Alfertshofer M, Hoch CC, Funk PF, Hollmann K, Wollenberg B, Knoedler S, et al. Sailing the seven seas: a multinational comparison of ChatGPT’s performance on medical licensing examinations. Ann Biomed Eng. 2023;52(6):1542–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Lourenco AP, Slanetz PJ, Baird GL. Rise of ChatGPT: it may be time to reassess how we teach and test radiology residents. Radiology. 2023;307(5):e231053. [DOI] [PubMed] [Google Scholar]
  • 53.Seetharaman R. Revolutionizing medical education: can ChatGPT boost subjective learning and expression? J Med Syst. 2023;47(1):61. [DOI] [PubMed] [Google Scholar]
  • 54.Horton JA, Ally I. Response to “Exploring the applications of ChatGPT in family medicine medical education.” PRiMER. 2023;7:28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Wójcik S, Rulkiewicz A, Pruszczyk P, Lisik W, Poboży M, Domienik-Karłowicz J. Reshaping medical education: performance of ChatGPT on a PES medical examination. Cardiol J. 2023;31(3):442–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Seth I, Xie Y, Hunter-Smith DJ, Seifman MA, Rozen WM. Re: Investigating the impact of innovative AI chatbot on post-pandemic medical education and clinical assistance. ANZ J Surg. 2023;94(1–2):68–77. [DOI] [PubMed] [Google Scholar]
  • 57.Webb JJ. Proof of concept: using ChatGPT to teach emergency physicians how to break bad news. Cureus. 2023;15(5): e38755. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Tsang R. Practical applications of ChatGPT in undergraduate medical education. J Med Educ Curric Dev. 2023;24(10):238212052311784. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Mirza FN, Lim RK, Yumeen S, Wahood S, Zaidat B, Shah A, et al. Performance of three large language models on dermatology board examinations. J Investig Dermatol. 2024;144(2):398–400. [DOI] [PubMed] [Google Scholar]
  • 60.Dhanvijay AKD, Pinjar MJ, Dhokane N, Sorte SR, Kumari A, Mondal H. Performance of large language models (ChatGPT, Bing Search, and Google Bard) in solving case vignettes in physiology. Cureus. 2023;15(8): e42972. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study. JMIR Med Educ. 2023;9:e48002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Borchert RJ, Hickman CR, Pepys J, Sadler TJ. Performance of ChatGPT on the situational judgement test—a professional dilemmas–based examination for doctors in the United Kingdom. JMIR Med Educ. 2023;7(9): e48978. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Flores-Cohaila JA, García-Vicente A, Vizcarra-Jiménez SF, De la Cruz-Galán JP, Gutiérrez-Arratia JD, Quiroga Torres BG, et al. Performance of ChatGPT on the Peruvian National Licensing Medical Examination: cross-sectional study. JMIR Med Educ. 2023;28(9): e48039. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Huang H. Performance of ChatGPT on registered nurse license exam in Taiwan: a descriptive study. Healthcare. 2023;11(21):2855. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Strong E, DiGiammarino A, Weng Y, Basaviah P, Hosamani P, Kumar A, et al. Performance of ChatGPT on free-response, clinical reasoning exams. medRxiv. 2023. [DOI] [PMC free article] [PubMed]
  • 66.Revercomb L, Patel AM, Choudhry HS, Filimonov A. Performance of ChatGPT in otolaryngology knowledge assessment. Am J Otolaryngol. 2024;45(1): 104082. [DOI] [PubMed] [Google Scholar]
  • 67.Levin G, Horesh N, Brezinov Y, Meyer R. Performance of <scp>ChatGPT</scp> in medical examinations: a systematic review and a meta-analysis. BJOG. 2024;131(3):378–80. [DOI] [PubMed] [Google Scholar]
  • 68.Ozeri DJ, Cohen A, Bacharach N, Ukashi O, Oppenheim A. Performance of ChatGPT in Israeli Hebrew Internal Medicine National Residency Exam. Isr Med Assoc J. 2024;26(2):86–8. [PubMed] [Google Scholar]
  • 69.Guigue P, Meyer R, Thivolle-Lioux G, Brezinov Y, Levin G. Performance of <scp>ChatGPT</scp> in French language Parcours d’Accès Spécifique Santé test and in <scp>OBGYN</scp>. Int J Gynecol Obstet. 2024;164(3):959–63. [DOI] [PubMed] [Google Scholar]
  • 70.Ali R, Tang OY, Connolly ID, Fridley JS, Shin JH, Zadnik Sullivan PL, et al. Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank. Neurosurgery. 2023;93(5):1090–8. [DOI] [PubMed] [Google Scholar]
  • 71.Torres-Zegarra BC, Rios-Garcia W, Ñaña-Cordova AM, Arteaga-Cisneros KF, Chalco XCB, Ordoñez MAB, et al. Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study. J Educ Eval Health Prof. 2023;20(20):30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72.Ali R, Tang OY, Connolly ID, Zadnik Sullivan PL, Shin JH, Fridley JS, et al. Performance of ChatGPT and GPT-4 on neurosurgery written board examinations. Neurosurgery. 2023;93(6):1353–65. [DOI] [PubMed] [Google Scholar]
  • 73.Fowler T, Pullen S, Birkett L. Performance of ChatGPT and Bard on the official part 1 FRCOphth practice questions. Br J Ophthalmol. 2023;108(10):1379–83. [DOI] [PubMed] [Google Scholar]
  • 74.Wang H, Wu W, Dou Z, He L, Yang L. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Inform. 2023;177: 105173. [DOI] [PubMed] [Google Scholar]
  • 75.Jowsey T, Stokes-Parish J, Singleton R, Todorovic M. Medical education empowered by generative artificial intelligence large language models. Trends Mol Med. 2023;29(12):971–3. [DOI] [PubMed] [Google Scholar]
  • 76.Kirpalani A, Grimmer J, Wang PZT. Med versus machine: Using ChatGPT in team-based learning. Med Educ. 2023;57(11):1159–60. [DOI] [PubMed] [Google Scholar]
  • 77.Mohapatra DP, Thiruvoth FM, Tripathy S, Rajan S, Vathulya M, Lakshmi P, et al. Leveraging large language models (LLM) for the plastic surgery resident training: do they have a role? Indian J Plast Surg. 2023;56(05):413–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 78.Gurnani B, Kaur K. Leveraging ChatGPT for ophthalmic education: a critical appraisal. Eur J Ophthalmol. 2024;34(2):323–7. [DOI] [PubMed] [Google Scholar]
  • 79.Abd-Alrazaq A, AlSaad R, Alhuwail D, Ahmed A, Healy PM, Latifi S, et al. Large language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023;1(9): e48291. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 80.Kumari A, Kumari A, Singh A, Singh SK, Juhi A, Dhanvijay AKD, et al. Large language models in hematology case solving: a comparative study of ChatGPT-3.5, Google Bard, and Microsoft Bing. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 81.Bair H, Norden J. Large language models and their implications on medical education. Acad Med. 2023;98(8):869–70. [DOI] [PubMed] [Google Scholar]
  • 82.Mannam SS, Subtirelu R, Chauhan D, Ahmad HS, Matache IM, Bryan K, et al. Large language model-based neurosurgical evaluation matrix: a novel scoring criteria to assess the efficacy of ChatGPT as an educational tool for neurosurgery board preparation. World Neurosurg. 2023;180:e765–73. [DOI] [PubMed] [Google Scholar]
  • 83.Waisberg E, Ong J, Masalkhi M, Lee AG. Large language model (LLM)-driven chatbots for neuro-ophthalmic medical education. Eye. 2023;38(4):639–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 84.Ghosh A, Maini Jindal N, Gupta VK, Bansal E, Kaur Bajwa N, Sett A. Is ChatGPT’s knowledge and interpretative ability comparable to first professional MBBS (Bachelor of Medicine, Bachelor of Surgery) students of india in taking a medical biochemistry examination? Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 85.Jamal A, Solaiman M, Alhasan K, Temsah MH, Sayed G. Integrating ChatGPT in medical education: adapting curricula to cultivate competent physicians for the AI era. Cureus. 2023;15(8): e43036. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 86.Haze T, Kawano R, Takase H, Suzuki S, Hirawa N, Tamura K. Influence on the accuracy in ChatGPT: differences in the amount of information per medical field. Int J Med Inform. 2023;180: 105283. [DOI] [PubMed] [Google Scholar]
  • 87.Teebagy S, Colwell L, Wood E, Yaghy A, Faustina M. Improved performance of ChatGPT-4 on the OKAP examination: a comparative study with ChatGPT-3.5. J Acad Ophthalmol. 2023;15(02):e184-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 88.Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;8(9): e45312. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 89.Alessandri Bonetti M, Giorgino R, Gallo Afflitto G, De Lorenzi F, Egro FM. How does ChatGPT perform on the Italian Residency Admission National Exam compared to 15,869 medical graduates? Ann Biomed Eng. 2023;52(4):745–9. [DOI] [PubMed] [Google Scholar]
  • 90.Haruna-Cooper L, Rashid MA. GPT-4: the future of artificial intelligence in medical school assessments. J R Soc Med. 2023;116(6):218–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 91.Guerra GA, Hofmann H, Sobhani S, Hofmann G, Gomez D, Soroudi D, et al. GPT-4 artificial intelligence model outperforms ChatGPT, medical students, and neurosurgery residents on neurosurgery written board-like questions. World Neurosurg. 2023;179:e160–5. [DOI] [PubMed] [Google Scholar]
  • 92.Moritz S, Romeike B, Stosch C, Tolks D. Generative AI (gAI) in medical education: Chat-GPT and co. GMS J Med Educ. 2023;40(4):Doc54. [DOI] [PMC free article] [PubMed]
  • 93.van de Ridder JMM, Shoja MM, Rajput V. Finding the place of ChatGPT in medical education. Acad Med. 2023;98(8):867–867. [DOI] [PubMed] [Google Scholar]
  • 94.Ayub I, Hamann D, Hamann CR, Davis MJ. Exploring the potential and limitations of chat generative pre-trained transformer (ChatGPT) in generating board-style dermatology questions: a qualitative analysis. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 95.Baglivo F, De Angelis L, Casigliani V, Arzilli G, Privitera GP, Rizzo C. Exploring the possible use of AI chatbots in public health education: feasibility study. JMIR Med Educ. 2023;1(9): e51421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 96.Morjaria L, Burns L, Bracken K, Ngo QN, Lee M, Levinson AJ, et al. Examining the threat of ChatGPT to the validity of short answer assessments in an undergraduate medical program. J Med Educ Curric Dev. 2023;28:10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 97.Rosoł M, Gąsior JS, Łaba J, Korzeniewski K, Młyńczak M. Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. Sci Rep. 2023;13(1):20512. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 98.Yaneva V, Baldwin P, Jurich DP, Swygert K, Clauser BE. Examining ChatGPT performance on USMLE sample items and implications for assessment. Acad Med. 2024;99(2):192–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 99.Lai UH, Wu KS, Hsu TY, Kan JKC. Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment. Front Med (Lausanne). 2023;19:10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 100.Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the performance of ChatGPT in ophthalmology. Ophthalmol Sci. 2023;3(4): 100324. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 101.Subramani M, Jaleel I, Krishna MS. Evaluating the performance of ChatGPT in medical physiology university examination of phase I MBBS. Adv Physiol Educ. 2023;47(2):270–1. [DOI] [PubMed] [Google Scholar]
  • 102.Giannos P. Evaluating the limits of AI in medical specialisation: ChatGPT’s performance on the UK Neurology Specialty Certificate Examination. BMJ Neurol Open. 2023;5(1): e000451. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 103.Guillen-Grima F, Guillen-Aguinaga S, Guillen-Aguinaga L, Alas-Brun R, Onambele L, Ortega W, et al. Evaluating the efficacy of ChatGPT in navigating the Spanish Medical Residency Entrance Examination (MIR): promising horizons for AI in clinical medicine. Clin Pract. 2023;13(6):1460–87. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 104.Zalzal HG, Cheng J, Shah RK. Evaluating the current ability of ChatGPT to assist in professional otolaryngology education. OTO Open. 2023;7(4): e94. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 105.Jiao C, Edupuganti NR, Patel PA, Bui T, Sheth V. Evaluating the artificial intelligence performance growth in ophthalmic knowledge. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 106.Ghosh A, Bir A. Evaluating ChatGPT’s ability to solve higher-order questions on the competency-based medical education curriculum in medical biochemistry. Cureus. 2023;15(4): e37023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 107.Agarwal M, Goswami A, Sharma P. Evaluating ChatGPT-3.5 and Claude-2 in answering and explaining conceptual medical physiology multiple-choice questions. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 108.Kung JE, Marshall C, Gauthier C, Gonzalez TA, Jackson JB. Evaluating ChatGPT performance on the orthopaedic in-training examination. JB JS Open Access. 2023;8(3):e23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 109.Beaulieu-Jones BR, Shah S, Berrigan MT, Marwaha JS, Lai SL, Brat GA. Evaluating capabilities of large language models: performance of GPT4 on surgical knowledge assessments. medRxiv. 2023. [DOI] [PMC free article] [PubMed]
  • 110.Ignjatović A, Stevanović L. Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study. J Educ Eval Health Prof. 2023;16(20):28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 111.Shay D, Kumar B, Redaelli S, von Wedel D, Liu M, Dershwitz M, et al. Could ChatGPT-4 pass an anaesthesiology board examination? Follow-up assessment of a comprehensive set of board examination practice questions. Br J Anaesth. 2024;132(1):172–4. [DOI] [PubMed] [Google Scholar]
  • 112.Friederichs H, Friederichs WJ, März M. ChatGPT in medical school: how successful is AI in progress testing? Med Educ Online. 2023;28(1):2220920. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 113.Lin JC, Younessi DN, Kurapati SS, Tang OY, Scott IU. Comparison of GPT-3.5, GPT-4, and human user performance on a practice ophthalmology written examination. Eye. 2023;37(17):3694–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 114.Massey PA, Montgomery C, Zhang AS. Comparison of ChatGPT–3.5, ChatGPT-4, and orthopaedic resident performance on orthopaedic assessment examinations. J Am Acad Orthop Surg. 2023;31(23):1173–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 115.Lin CC, Akuhata-Huntington Z, Hsu CW. Comparing ChatGPT’s ability to rate the degree of stereotypes and the consistency of stereotype attribution with those of medical students in New Zealand in developing a similarity rating test: a methodological study. J Educ Eval Health Prof. 2023;12(20):17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 116.Hoch CC, Wollenberg B, Lüers JC, Knoedler S, Knoedler L, Frank K, et al. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. 2023;280(9):4271–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 117.Lower K, Seth I, Lim B, Seth N. ChatGPT-4: transforming medical education and addressing clinical exposure challenges in the post-pandemic Era. Indian J Orthop. 2023;57(9):1527–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 118.Liu X, Wu C, Lai R, Lin H, Xu Y, Lin Y, et al. ChatGPT: when the artificial intelligence meets standardized patients in clinical training. J Transl Med. 2023;21(1):447. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 119.Cheung BHH, Lau GKK, Wong GTC, Lee EYP, Kulkarni D, Seow CS, et al. ChatGPT versus human in generating medical graduate exam multiple choice questions—a multinational prospective study (Hong Kong S.A.R, Singapore, Ireland, and the United Kingdom). PLoS One. 2023;18(8):e0290691. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 120.Skalidis I, Cagnina A, Luangphiphat W, Mahendiran T, Muller O, Abbe E, et al. ChatGPT takes on the European Exam in Core Cardiology: an artificial intelligence success story? Eur Heart J-Digit Health. 2023;4(3):279–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 121.Gandhi Periaysamy A, Satapathy P, Neyazi A, Padhi BK. ChatGPT: roles and boundaries of the new artificial intelligence tool in medical education and health research - correspondence. Ann Med Surg (Lond). 2023;85(4):1317–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 122.Wang X, Gong Z, Wang G, Jia J, Xu Y, Zhao J, et al. ChatGPT performs on the Chinese National Medical Licensing Examination. J Med Syst. 2023;47(1):86. [DOI] [PubMed] [Google Scholar]
  • 123.Oztermeli AD, Oztermeli A. ChatGPT performance in the medical specialty exam: an observational study. Medicine. 2023;102(32): e34673. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 124.Meo SA, Al-Masri AA, Alotaibi M, Meo MZS, Meo MOS. ChatGPT knowledge evaluation in basic and clinical medical sciences: multiple choice question examination-based performance. Healthcare. 2023;11(14):2046. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 125.Scherr R, Halaseh FF, Spina A, Andalib S, Rivera R. ChatGPT interactive medical simulations for early clinical education: case study. JMIR Med Educ. 2023;10(9): e49877. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 126.Sabry Abdel-Messih M, Kamel Boulos MN. ChatGPT in clinical toxicology. JMIR Med Educ. 2023;8(9): e46876. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 127.Oh N, Choi GS, Lee WY. ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models. Ann Surg Treat Res. 2023;104(5):269–73. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 128.Weng TL, Wang YM, Chang S, Chen TJ, Hwang SJ. ChatGPT failed Taiwan’s family medicine board exam. J Chin Med Assoc. 2023;86(8):762–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 129.Aljindan FK, Al Qurashi AA, Albalawi IAS, Alanazi AMM, Aljuhani HAM, Falah Almutairi F, et al. ChatGPT Conquers the Saudi Medical Licensing Exam: exploring the accuracy of artificial intelligence in medical knowledge assessment and implications for modern medical education. Cureus. 2023;15(9): e45043. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 130.Sodhi KS, Tao TY, Seymore N. ChatGPT: chasing the storm in radiology training and education. Indian J Radiol Imaging. 2023;33(4):431–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 131.Zhu L, Mou W, Yang T, Chen R. ChatGPT can pass the AHA exams: open-ended questions outperform multiple-choice format. Resuscitation. 2023;188: 109783. [DOI] [PubMed] [Google Scholar]
  • 132.Feng S, Shen Y. ChatGPT and the future of medical education. Acad Med. 2023;98(8):867–8. [DOI] [PubMed] [Google Scholar]
  • 133.Kumah-Crystal Y, Mankowitz S, Embi P, Lehmann CU. ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification? J Am Med Inform Assoc. 2023;30(9):1558–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 134.Boscardin CK, Gin B, Golde PB, Hauer KE. ChatGPT and generative artificial intelligence for medical education: potential impact and opportunity. Acad Med. 2024;99(1):22–7. [DOI] [PubMed] [Google Scholar]
  • 135.Strong E, DiGiammarino A, Weng Y, Kumar A, Hosamani P, Hom J, et al. Chatbot vs medical student performance on free-response clinical reasoning examinations. JAMA Intern Med. 2023;183(9):1028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 136.Suchman K, Garg S, Trindade AJ. Chat generative pretrained transformer fails the multiple-choice American College of Gastroenterology Self-Assessment Test. Am J Gastroenterol. 2023;118(12):2280–2. [DOI] [PubMed] [Google Scholar]
  • 137.Antaki F, Milad D, Chia MA, Giguère CÉ, Touma S, El-Khoury J, et al. Capabilities of GPT-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering. Br J Ophthalmol. 2023;108(10):1371–8. [DOI] [PubMed] [Google Scholar]
  • 138.Gencer A, Aydin S. Can ChatGPT pass the thoracic surgery exam? Am J Med Sci. 2023;366(4):291–5. [DOI] [PubMed] [Google Scholar]
  • 139.Shang L, Xue M, Hou Y, Tang B. Can ChatGPT pass China’s national medical licensing examination? Asian J Surg. 2023;46(12):6112–3. [DOI] [PubMed] [Google Scholar]
  • 140.Leung TI, Sagar A, Shroff S, Henry TL. Can AI mitigate bias in writing letters of recommendation? JMIR Med Educ. 2023;23(9): e51494. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 141.Huang Y, Gomaa A, Semrau S, Haderlein M, Lettmaier S, Weissmann T, et al. Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology. Front Oncol. 2023;14:13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 142.Huang RS, Lu KJQ, Meaney C, Kemppainen J, Punnett A, Leung FH. Assessment of resident and AI chatbot performance on the University of Toronto Family Medicine Residency Progress Test: comparative study. JMIR Med Educ. 2023;19(9): e50514. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 143.Chen TC, Multala E, Kearns P, Delashaw J, Dumont A, Maraganore D, et al. Assessment of ChatGPT’s performance on neurology written board examination questions. BMJ Neurol Open. 2023;5(2): e000530. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 144.Shay D, Kumar B, Bellamy D, Palepu A, Dershwitz M, Walz JM, et al. Assessment of ChatGPT success with specialty medical knowledge using anaesthesiology board examination practice questions. Br J Anaesth. 2023;131(2):e31–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 145.Biri SK, Kumar S, Panigrahi M, Mondal S, Behera JK, Mondal H. Assessing the utilization of large language models in medical education: insights from undergraduate medical students. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 146.Surapaneni KM. Assessing the performance of ChatGPT in medical biochemistry using clinical case vignettes: observational study. JMIR Med Educ. 2023;7(9): e47191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 147.Chen J, Cadiente A, Kasselman LJ, Pilkington B. Assessing the performance of ChatGPT in bioethics: a large language model’s moral compass in medicine. J Med Ethics. 2024;50(2):97–101. [DOI] [PubMed] [Google Scholar]
  • 148.Banerjee A, Ahmad A, Bhalla P, Goyal K. Assessing the efficacy of ChatGPT in solving questions based on the core concepts in physiology. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 149.Das D, Kumar N, Longjam LA, Sinha R, Deb Roy A, Mondal H, et al. Assessing the capability of ChatGPT in answering first- and second-order knowledge questions on microbiology as per competency-based medical education curriculum. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 150.Johnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, et al. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Res Sq. 2023.
  • 151.Sallam M, Salim NA, Barakat M, Al-Mahzoum K, Al-Tammemi AB, Malaeb D, et al. Assessing health students’ attitudes and usage of ChatGPT in Jordan: validation study. JMIR Med Educ. 2023;5(9): e48254. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 152.Saad A, Iyengar KP, Kurisunkal V, Botchu R. Assessing ChatGPT’s ability to pass the FRCS orthopaedic part A exam: a critical analysis. The Surgeon. 2023;21(5):263–6. [DOI] [PubMed] [Google Scholar]
  • 153.Johnstone RE, Neely G, Sizemore DC. Artificial intelligence software can generate residency application personal statements that program directors find acceptable and difficult to distinguish from applicant compositions. J Clin Anesth. 2023;89: 111185. [DOI] [PubMed] [Google Scholar]
  • 154.Moshirfar M, Altaf AW, Stoakes IM, Tuttle JJ, Hoopes PC. Artificial intelligence in ophthalmology: a comparative analysis of GPT-3.5, GPT-4, and human expertise in answering StatPearls questions. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 155.Roos J, Kasapovic A, Jansen T, Kaczmarczyk R. Artificial intelligence in medical education: comparative analysis of ChatGPT, Bing, and medical students in Germany. JMIR Med Educ. 2023;4(9): e46482. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 156.Huh S. Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study. J Educ Eval Health Prof. 2023;11(20):1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 157.Gupta R, Park JB, Herzog I, Yosufi N, Mangan A, Firouzbakht PK, et al. Applying GPT-4 to the plastic surgery inservice training examination. J Plast Reconstr Aesthet Surg. 2023;87:78–82. [DOI] [PubMed] [Google Scholar]
  • 158.Pendergrast T, Chalmers Z. Anki Tagger: a generative AI tool for aligning third-party resources to preclinical curriculum. JMIR Med Educ. 2023;20(9): e48780. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 159.Sanna Reddy J, Padmakumar Usha A, Appavu R, Surapaneni KM. Analyzing the surgical knowledge of ChatGPT in undergraduate written medical examination. Indian J Surg. 2023;85(6):1524–6. [Google Scholar]
  • 160.Sivasubramanian J, Shaik Hussain SM, Virudhunagar Muthuprakash S, Periadurai ND, Mohanram K, Surapaneni KM. Analysing the clinical knowledge of ChatGPT in medical microbiology in the undergraduate medical examination. Indian J Med Microbiol. 2023;45: 100380. [DOI] [PubMed] [Google Scholar]
  • 161.Agarwal M, Sharma P, Goswami A. Analysing the applicability of ChatGPT, Bard, and Bing to generate reasoning-based multiple-choice questions in medical physiology. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 162.Han Z, Battaglia F, Udaiyar A, Fooks A, Terlecky SR. An explorative assessment of ChatGPT as an aid in medical education: use it with caution. Med Teach. 2023;20:1–8. [DOI] [PubMed] [Google Scholar]
  • 163.Ghorashi N, Ismail A, Ghosh P, Sidawy A, Javan R. AI-powered chatbots in medical education: potential applications and implications. Cureus. 2023. [DOI] [PMC free article] [PubMed]
  • 164.Klang E, Portugez S, Gross R, Kassif Lerner R, Brenner A, Gilboa M, et al. Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4. BMC Med Educ. 2023;23(1):772. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 165.Yanagita Y, Yokokawa D, Uchida S, Tawara J, Ikusaka M. Accuracy of ChatGPT on medical questions in the National Medical Licensing Examination in Japan: evaluation study. JMIR Form Res. 2023;13(7): e48023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 166.Ishaaq N, Sohail SS. Correspondence to investigating the impact of innovative AI chatbot on post-pandemic medical education and clinical assistance: a comprehensive analysis. ANZ J Surg. 2023;94(3):494. [DOI] [PubMed] [Google Scholar]
  • 167.Lindeman BM, Lipsett PA. Evaluation and feedback. In: Thomas PA, Kern DE, Hughes MT, Tackett SA, Chen BY, editors. Curriculum development for medical education: a six-step approach. JHU press; 2016. p. 122–67. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.


Articles from Medical Science Educator are provided here courtesy of Springer

RESOURCES