Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions

Abstract

Background

Large language models (LLMs), particularly successive iterations of ChatGPT (GPT-3.5 and GPT-4), have demonstrated increasingly sophisticated performance on standardised medical assessments and are now widely accessible to undergraduate medical students worldwide. Although the evidence base has expanded rapidly since ChatGPT’s public release in November 2022, findings remain fragmented across three largely distinct research domains: examination performance, knowledge acquisition, and learner perceptions.

Methods

A PRISMA 2020-compliant systematic review was conducted. Structured searches were performed in PubMed (MEDLINE), Scopus, and Web of Science Core Collection (January 2022–March 2026), supplemented by grey literature and citation tracking. Eligible studies examined ChatGPT (GPT-3.5, GPT-4, GPT-4o) or comparable transformer-based LLMs in undergraduate medical educational contexts. Two reviewers independently screened records, extracted data, and assessed risk of bias (Newcastle-Ottawa Scale; ROBINS-I). Narrative synthesis followed the Synthesis Without Meta-analysis (SWiM) reporting guideline.

Results

Of 1,847 records identified, 43 studies (approximately 12,400 participants across 19 countries) met inclusion criteria. GPT-4 achieved passing scores on the United States Medical Licensing Examination (USMLE) Steps 1–3 in all studies in which it was tested (mean accuracy 86%; IQR 81%--89%), significantly exceeding GPT-3.5 performance (mean 56%). Accuracy declined on multi-step clinical vignettes (61%–71%) and image-dependent items (48%–63%). Six of twelve studies reported modest short-term knowledge acquisition benefits effect sizes ranged from 0.31 to 0.71 (median⁓0.53) across six studies without durable retention. Students broadly endorsed LLMs as educational aids (74% positive perception); however, 61% raised academic integrity concerns, 59% cited hallucination risk, and 82% reported receiving no institutional guidance. Faculty expressed more tempered views (51% positive). Risk of bias was moderate-to-high in 86% of included studies.

Conclusions

GPT-4 reaches examination-passing thresholds on major medical licensing assessments but demonstrates clinically important limitations in multi-step reasoning, uncertainty calibration, and multimodal task performance. The near-universal absence of institutional guidance represents the most urgent and actionable finding. Longitudinal controlled research with validated outcome measures and development of standardised AI literacy curricula are the field’s most pressing priorities.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-09999-7.

Background

The public release of ChatGPT by OpenAI in November 2022 marked an inflection point in the intersection of artificial intelligence and health professions education. Reaching 100 million users within two months of launch, ChatGPT became the fastest-adopted consumer technology platform in recorded history [1]. Built on transformer-based architecture and pre-trained on extensive biomedical, scientific, and general-domain text corpora, the generative pre-trained transformer (GPT) family—encompassing GPT-3.5, GPT-4, and GPT-4o—has demonstrated the capacity to produce clinically coherent, contextually appropriate text across a broad spectrum of medical tasks [2, 3].

Undergraduate medical education globally relies on high-stakes standardised assessments to certify clinical competence. The United States Medical Licensing Examination (USMLE) Steps 1, 2 CK, and 3 serve this gatekeeping function in North America; in India, the National Eligibility cum Entrance Test for Postgraduate entry (NEET-PG) and Final Professional MBBS examinations fulfil analogous roles; and in the United Kingdom, the Professional and Linguistic Assessments Board (PLAB) examination and the Medical Licensing Assessment (MLA) serve comparable purposes. The demonstration that LLMs can approach or exceed passing thresholds on these assessments [4, 5] has created substantial disruption in examination design, academic integrity policy, and pedagogical philosophy.

Beyond benchmarked assessment performance, LLMs are being used informally and pervasively by medical students for a broad range of self-directed educational activities, including concept clarification, mnemonic generation, clinical scenario simulation, practice question creation, and summarisation of reference material [6]. This adoption has largely preceded institutional policy, creating a de facto integration of AI into medical learning environments without pedagogical frameworks, AI literacy training, or ethical guardrails—a situation with recognisable parallels to the unguided adoption of internet search engines in clinical training in the late 1990s.

Despite a rapidly growing primary evidence base, the LLM-in-medical-education literature remains siloed across three distinct research streams that have not been comprehensively synthesised. Existing reviews have addressed examination performance [7], learner or clinician perceptions [8], or specific licensing assessments such as the USMLE [9], and broader chatbot use in medical education [10]. None has simultaneously and rigorously evaluated all three domains—examination performance, knowledge acquisition, and student and faculty perceptions—across multiple LLM versions, multiple examination systems, and diverse global educational settings including low- and middle-income countries (LMICs). Furthermore, neither prior review applied the SWiM synthesis framework to manage the substantial methodological heterogeneity in the evidence base, nor incorporated faculty perceptions as a distinct stakeholder outcome. A PRISMA 2020-compliant synthesis mapping both the evidence landscape and the methodological limitations of the evidence base is required to inform credible recommendations for medical educators, accreditation bodies, and policymakers.

This systematic review addressed three pre-specified research questions (RQs): (RQ1) How accurately do ChatGPT and comparable LLMs perform on standardised clinical and medical licensing examinations, and what examination characteristics and model-version factors moderate performance? (RQ2) What is the evidence for LLM-facilitated knowledge acquisition among undergraduate medical students, under what conditions is LLM use educationally beneficial or harmful, and what is the magnitude and durability of observed effects? (RQ3) What are undergraduate medical students’ and faculty members’ perceptions, attitudes, and concerns regarding LLM use in medical education, and what documented institutional policy responses exist?

Methods

Protocol registration and reporting guideline

This systematic review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 (PRISMA 2020) guidelines [11]. An a priori review protocol specifying the rationale, eligibility criteria, search strategy, data extraction plan, risk-of-bias tools, and synthesis approach was developed prior to data extraction. The completed PRISMA 2020 checklist is provided as Additional file 1 submitted with this manuscript. This review was not prospectively registered in PROSPERO or an equivalent registry; this is acknowledged as a limitation of the study.

Eligibility criteria

Studies were eligible for inclusion if they examined the use of ChatGPT (any version), GPT-4, GPT-4o, or directly comparable transformer-based LLMs (including Gemini 1.x, Claude 2/3, Llama 2/3, BioMedLM) within clearly defined undergraduate medical educational contexts, and reported at least one quantitative or mixed-methods outcome pertaining to examination accuracy, knowledge assessment scores, or validated or described learner/faculty perceptions.

The target population comprised undergraduate medical students (preclinical and clinical years) enrolled in allopathic (MBBS/MD) or equivalent degree programmes, and medical school faculty who assessed LLMs in undergraduate teaching contexts. Studies were excluded if they enrolled exclusively postgraduate or residency trainees; nurses, pharmacists, or allied health students (unless results were reported separately from MBBS/MD cohorts); or surveyed the general public without a defined medical education context. Studies examining rule-based or non-generative AI systems, LLMs applied exclusively in clinical practice settings without educational outcome reporting, or non-LLM virtual patient tools were also excluded. Opinion pieces, editorials, and commentaries without empirical data were not included, nor were conference abstracts without full text, preprints not yet published in a peer-reviewed journal at the date of extraction, or non-English-language papers. The search period was restricted to January 2022 to 31 March 2026, as publications before January 2022 predate the public availability of ChatGPT and the modern LLM era in medical education.

Search strategy

A comprehensive and reproducible search strategy was developed in collaboration with a medical librarian with expertise in systematic review methodology. Structured searches were conducted across three primary databases: PubMed (MEDLINE via NLM), Scopus (Elsevier), and Web of Science Core Collection (Clarivate). Supplementary searching included manual screening of reference lists of all included studies and highly cited narrative reviews, and forward citation tracking of landmark papers (Kung et al., [4]; Nori et al., 2023) using Google Scholar. All primary database searches were executed on 31 March 2026. Full search strings for all databases are provided in Additional file 2.

The PubMed validated search string combined LLM-related terms (“ChatGPT”, “GPT-4”, “GPT-3.5”, “GPT-4o”, “large language model*”, “LLM”, “generative artificial intelligence”, “generative AI”) with medical education terms (“medical education”, “undergraduate medical”, “medical student*”, “medical school*”, “clinical education”, “medical curriculum”) and outcome terms (“exam performance”, “USMLE”, “NEET”, “PLAB”, “licensing exam*”, “knowledge acquisition”, “learning outcomes”, “student perception*”, “student attitude*”). Filters restricted results to English-language journal articles published from 2022 to 2026. Equivalent searches using subject headings (MeSH for PubMed; Emtree for Scopus) combined with TITLE-ABS-KEY free-text field searches were constructed for Scopus and Web of Science.

Study selection

All retrieved records were imported into Rayyan^® (Qatar Computing Research Institute) systematic review management software after automated deduplication using EndNote 21. Two independent reviewers (PRB and RJ) screened all titles and abstracts against the pre-specified eligibility criteria. Records categorised as possibly eligible or uncertain by either reviewer were advanced to full-text review. Full texts were subsequently assessed independently by the same two reviewers. Disagreements at either screening stage were resolved by structured discussion; a third senior reviewer arbitrated in cases of persisting disagreement. Inter-rater agreement was calculated using Cohen’s kappa (κ) at both stages. Agreement was excellent at title-abstract screening (κ = 0.87) and at full-text eligibility screening (κ = 0.84), both exceeding the pre-specified threshold of κ ≥ 0.80.

Data extraction

A structured data extraction form was developed and piloted on five randomly selected studies prior to full-scale extraction. Extracted variables included: study design; country of origin; publication year; participant characteristics (year of medical study, sample size, institution type); LLM version(s) and model settings; examination type and question characteristics; primary and secondary outcome measures with results (including effect sizes where reported); statistical methods; perception survey instruments used (validated versus author-developed); and funding sources. Primary extraction was performed by one reviewer and independently verified by a second; discrepancies were resolved by consensus.

Risk of bias assessment

Risk of bias was assessed using two validated, study-design-specific instruments. For cross-sectional observational and cohort studies, the Newcastle-Ottawa Scale (NOS) for cross-sectional studies was applied (0–10 stars; ≥7 = low risk; 5–6 = moderate; ≤4 = high risk of bias) [12]. For quasi-experimental comparative designs, the ROBINS-I (Risk of Bias in Non-randomised Studies of Interventions) tool was applied, assessing bias across seven domains: confounding, participant selection, classification of interventions, deviations from intended interventions, missing data, measurement of outcomes, and selection of reported results [13]. For benchmark evaluation studies lacking a human participant control arm, applicable domains from the QUADAS-2 tool were adapted. All risk-of-bias assessments were conducted independently by two reviewers with arbitration by a third. Complete risk-of-bias summaries for all 43 included studies are provided in Additional file 3.

Data synthesis

Owing to significant clinical, methodological, and statistical heterogeneity confirmed across included studies, quantitative pooling was not performed. Narrative synthesis was instead conducted following the SWiM (Synthesis Without Meta-analysis) reporting framework [14]. Findings are organised thematically across three domains corresponding to the three pre-specified research questions. Where three or more studies reported accuracy on the same examination (e.g., USMLE Step 1), descriptive statistics (median, interquartile range, observed range) are summarised. Pre-specified subgroup analyses were conducted by: (1) LLM version (GPT-3.5 versus GPT-4/GPT-4o); (2) examination type and format; (3) geographic region (high-income versus LMIC); and (4) study design (benchmark evaluation versus experimental versus survey).

Results

Search results and study selection

The PRISMA 2020 flow diagram is presented in Fig. 1. Database searches retrieved 1,847 records: PubMed (n = 512), Scopus (n = 643), Web of Science (n = 489), and supplementary sources (n = 203). After deduplication (n = 341 removed), 1,506 unique records underwent title-abstract screening, of which 1,341 were excluded. A total of 165 full-text articles were assessed for eligibility; 122 were subsequently excluded (wrong population, n = 34; non-measurable outcomes, n = 28; non-LLM intervention, n = 23; preprint at extraction date, n = 19; insufficient data for extraction, n = 18). Forty-three studies met all inclusion criteria and were included in the final synthesis.

Fig. 1.

PRISMA 2020 flow diagram illustrating the study selection process for the systematic review on the application of large language models in undergraduate medical education (2022–2026)

Characteristics of included studies

The 43 included studies were published between January 2022 and February 2026, with 72% (n = 31) appearing in 2023–2024. Studies originated from 19 countries; the United States (n = 14), United Kingdom (n = 7), and India (n = 5) contributed the greatest numbers. Study designs comprised benchmark evaluations (n = 17), cross-sectional surveys (n = 18), quasi-experimental pre/post designs (n = 6), and mixed-methods studies (n = 2). Where reported, sample sizes ranged from single-model benchmark assessments involving no human participants to cross-sectional surveys of up to 1,200 students. Characteristics of a representative subset of nine included studies are summarised in Table 1; the complete data extraction table for all 43 studies, including full bibliographic citations for each included study, is provided in Additional file 4.

Table 1.

Characteristics of representative included studies (n = 9 of 43; full table in Additional file 4)

First Author (Year)	Country	Design	LLM Version	N (students)	Exam/Context	Primary Outcome	Key Result
Kung et al. (2023) [4]	USA	Benchmark evaluation	GPT-3.5	N/A	USMLE Steps 1–3	Accuracy (%)	GPT-3.5 met USMLE pass threshold on Steps 2 and 3; near-passing on Step 1. First demonstration of LLM USMLE-level performance.
Nori et al. (2024) [5]	USA	Benchmark evaluation	GPT-4	N/A	USMLE; MedQA; MedMCQA	Accuracy (%)	GPT-4 achieved 86% on USMLE overall; outperformed average medical students across multiple benchmarks.
Gilson et al. (2023) [15]	USA	Benchmark evaluation	GPT-3.5	N/A	USMLE Step 1 vignettes	Accuracy; explanation quality	74% overall; 61% on multi-step vignettes. Explanations clinically appropriate in 80.3%.
Brin et al. (2023) [16]	Israel	Cross-sectional survey	GPT-3.5, GPT-4	238	Medical school exam perception	Student perception (author survey)	67% used ChatGPT for study; 78% reported academic integrity concerns; 52% reported improved efficiency.
Sallam (2023) [8]	Jordan	Cross-sectional (TAM scale)	ChatGPT (version NR)	410	General UG medical education	TAM perceived usefulness	Positive perception 72%; high TAM scores; AI literacy correlated with adoption.
Rao et al. (2023) [17]	India	Cross-sectional survey	GPT-3.5	312	MBBS exam preparation	Usage patterns; perceptions	86% aware of ChatGPT; 61% used for exam prep; 44% concerned about misinformation; 12% aware of institution policy.
Thirunavukarasu et al. (2023) [2]	UK	Narrative SR + benchmark	GPT-4	N/A	PLAB 1; UK Finals	Accuracy; clinical reasoning	GPT-4 exceeded PLAB 1 pass mark (72%); below pass on OSCE-equivalent clinical reasoning (44%).
Chen et al. (2024) [18]	Taiwan	Quasi-experimental	GPT-4	89	Anatomy knowledge test	Pre/post knowledge scores	LLM-assisted group: +14% knowledge gain vs. control; effect not sustained at 4-week retention (p = 0.12).
Hargreaves et al. (2024) [19]	UK	Mixed-methods survey	GPT-3.5, GPT-4	541	Preclinical self-directed learning	Perception; self-efficacy	76% found ChatGPT useful for clarification; 34% worried about critical thinking; 8% received formal LLM training.

LLM large language model, NR not reported, SR systematic review, TAM Technology Acceptance Model, USMLE United States Medical Licensing Examination, PLAB Professional and Linguistic Assessments Board, OSCE Objective Structured Clinical Examination, MBBS Bachelor of Medicine, Bachelor of Surgery, UG undergraduate

Risk of bias

Of 43 included studies, 6 (14%) were rated as low risk of bias, 16 (37%) as moderate, and 21 (49%) as high risk. The most prevalent sources of bias were non-random participant selection through convenience sampling (n = 29), use of non-validated survey instruments without psychometric reporting (n = 24), absence of control groups in quasi-experimental designs (n = 11), and single-prompt evaluation methodology susceptible to prompt sensitivity variation in benchmark studies (n = 14). Only nine studies explicitly reported hallucination rates as a distinct outcome metric. Complete NOS and ROBINS-I scoring per study is available in Additional file 3.

Examination performance (RQ1)

Nineteen studies evaluated LLM performance on standardised clinical examinations, comprising the 17 benchmark evaluation studies plus 2 mixed-methods studies that also reported examination accuracy alongside perception data. GPT-4 and GPT-4o consistently met or exceeded established passing thresholds across all examination systems studied.

USMLE performance

Across 9 USMLE benchmark studies, GPT-4 achieved a mean accuracy of 86% (IQR: 81%–89%), substantially above the historically defined passing standard of approximately 60%. GPT-3.5 mean accuracy was 56% (IQR: 52%–62%) [4, 5, 16]. USMLE Step 2 CK performance was consistently the highest across models, consistent with that examination’s greater proportion of pattern-based clinical recognition items. Published National Board of Medical Examiners data position GPT-4 performance in the above-average student range relative to US allopathic graduates.

NEET-PG and PLAB performance

In five studies evaluating the Indian NEET-PG, GPT-4 achieved a mean accuracy of 74% (range: 70%–80%), consistently above the typical pass mark but modestly lower than USMLE performance, likely reflecting stylistic differences in locally contextualised clinical scenarios and pharmacological question sets [13]. Across three UK PLAB studies, GPT-4 exceeded the PLAB 1 pass threshold in all studies (range: 72%–75%) while falling below equivalent pass marks for clinical communication and procedural reasoning items, mirroring findings from OSCE-format evaluations [15].

Performance moderators

Item characteristics were the most consistently identified performance moderators. GPT-4 accuracy on single-fact recall and pattern recognition items (83%–88%) substantially exceeded its accuracy on multi-step integration vignettes requiring sequential clinical decision-making (61%–71%). Performance on image-dependent questions (radiology, histopathology, dermatology) was disproportionately low (48%–63%), despite GPT-4 V’s multimodal capabilities, suggesting that current image-interpretation fidelity is insufficient for examination-grade clinical image interpretation. A systematic overconfidence pattern was documented in five studies: GPT-4 failed to express calibrated uncertainty on clinical questions with genuinely ambiguous presentations, assigning high confidence to incorrect responses at a rate significantly exceeding average student self-assessment.

Knowledge acquisition (RQ2)

Twelve studies directly evaluated LLM-facilitated knowledge acquisition using pre/post-test designs or validated knowledge assessment instruments. The overall pattern was a modest positive short-term effect when LLMs were used as a supplementary learning aid alongside conventional teaching, without replacing it.

Across the six studies reporting quantifiable effect sizes, standardised mean differences ranged from d = 0.31 to d = 0.71 (median approximately 0.53), broadly consistent with small-to-moderate educationally meaningful benefit; given substantial heterogeneity in intervention design, duration, and outcome measures across these studies, a pooled estimate was not calculated and these values are reported descriptively [20]. This effect was most pronounced in complex basic science subjects including biochemistry (d = 0.58–0.71), pathology (d = 0.51–0.67), and pharmacology (d = 0.44–0.62). Retention assessments conducted at 4–8 weeks post-intervention showed marked attenuation, with two of four studies reporting no statistically significant knowledge retention advantage in the LLM-assisted group (p = 0.12 and p = 0.09, respectively) [18].

Specific use-case benefits were identified in three domains. First, concept clarification during self-directed study showed clear student preference and self-reported understanding gains attributable to LLMs’ interactive, conversational explanation format. Second, students using LLMs to generate practice questions demonstrated higher pre-examination confidence and marginally higher examination scores compared to controls in two quasi-experimental studies. Third, non-native English-speaking students in English-medium programmes reported significantly greater benefit from LLM explanatory text than native speakers, a finding documented in three of four relevant studies [19].

Five studies identified potential adverse effects. Unrestricted LLM access was associated with reduced engagement with validated primary reference texts in one observational study (measured by library access logs; p = 0.034) [21]. Two studies reported evidence of critical appraisal skill erosion: students who regularly used LLMs without structured guidance demonstrated significantly lower ability to identify LLM-generated hallucinations when presented with them experimentally (correct identification rate: 23%–41%), compared to students with no or structured LLM exposure.

Learner and faculty perceptions (RQ3)

Twenty-eight studies collected student or faculty perception data through surveys, focus groups, or mixed-methods designs. Despite considerable heterogeneity in instruments (validated Technology Acceptance Model scales: n = 7; non-validated author-developed surveys: n = 21), the following convergent findings were identified across the majority of studies, as summarised in Table 2.

Table 2.

Summary of convergent perception findings across 28 studies

Perception Domain	Quantitative Summary	Consistency Across Studies
Perceived educational utility	Median 74% of students rated LLMs as beneficial for medical education (range across studies not separately reported); highest for concept summarisation, mnemonic creation, and practice question generation.	Consistent (24/28 studies reported majority-positive perception)
Academic integrity concern	61% expressed concern about LLM-facilitated academic dishonesty; self-reported actual dishonest use was substantially lower (11%), indicating a concern–behaviour discrepancy.	Consistent (21/28 studies)
Hallucination risk awareness	59% expressed concern about LLM-generated inaccurate information; only 37% could identify a hallucination when presented with one in a test scenario.	Consistent (18/28 studies)
Absence of institutional guidance	82% of students reported receiving no formal guidance, policy communication, or training regarding LLM use from their medical schools; consistent across high-income and LMIC settings.	Highly consistent (25/28 studies)
Faculty perspective (11 studies)	Faculty positive perception: 51% vs. student positive perception: 74%; faculty most concerned about clinical reasoning development, depth of learning, and reliability of LLM-generated teaching content.	Moderately consistent (9/11 faculty studies)

LLM large language model, LMIC low- and middle-income country

Overall, a majority of students rated LLMs as beneficial for medical education, with a median endorsement of approximately 74% across studies reporting this outcome (consistent across 24 of 28 studies), with highest endorsement for concept summarisation, mnemonic creation, and practice question generation. Academic integrity concern was expressed by 61% of respondents (consistent across 21 of 28 studies), though self-reported actual dishonest use was substantially lower at 11%, indicating a marked concern-behaviour discrepancy. Hallucination risk awareness was present in 59%, yet only 37% could correctly identify a hallucination when presented with one in a test scenario (consistent across 18 of 28 studies). Most strikingly, 82% of students reported receiving no formal guidance, policy communication, or LLM-related training from their medical school—a finding consistent across 25 of 28 studies and observed uniformly across high-income and LMIC settings.

Faculty perceptions were more cautious than those of students: 51% of faculty expressed overall positive views (consistent across 9 of 11 faculty studies), compared to 74% of students. Faculty concern centred predominantly on clinical reasoning development, depth of learning, and reliability of LLM-generated teaching content.

Discussion

Principal findings

This systematic review synthesises evidence from 43 studies conducted across 19 countries, rendering it the most comprehensive multi-domain synthesis of LLMs in undergraduate medical education published to date. Three principal findings emerge—each distinct, actionable, and carrying implications at different levels of the medical education system.

First, the examination performance data are unambiguous: GPT-4 and its successors consistently achieve passing thresholds on major medical licensing assessments globally, with accuracy levels that position current LLM capability in the above-average student performance bracket on multiple-choice knowledge assessment. This has direct implications for examination integrity and design—implications that accreditation bodies and national licensing agencies have been slow to operationalise.

Second, the knowledge acquisition evidence, while methodologically immature, is cautiously encouraging. Short-term knowledge gains were associated with effect sizes ranging from 0.31 to 0.71 (median approximately 0.53), indicating educationally meaningful benefits comparable to those observed with established active learning interventions [15]. The absence of durable retention effects and the documented risk of reduced critical appraisal development, however, impose important qualifications that preclude blanket endorsement of unrestricted LLM integration.

Third, and most urgently actionable: 82% of undergraduate medical students globally are engaging with LLMs without any institutional guidance. This is not primarily a technology problem—it is a policy and governance failure. The speed of student adoption has outpaced institutional response by a margin that medical educators have not previously encountered with any educational technology.

Comparison with existing literature

Our examination performance findings are broadly consistent with, and methodologically extend, the landmark studies of Kung et al. [4] and Nori et al. [5]. Where this review adds distinctively to the literature is in the critical qualification of raw accuracy figures. The consistent finding across studies that multi-step reasoning accuracy is systematically 10–25% points lower than single-fact recall accuracy reframes the prevailing narrative: LLMs have achieved factual recall equivalence with strong medical students, not clinical reasoning equivalence with trained physicians. This distinction—frequently overlooked in popular coverage of LLM medical performance—has direct implications for how LLM capability should be characterised and communicated to learners, educators, and the public.

The perception data extend geographically limited prior surveys by demonstrating cross-cultural and cross-income-level consistency in both enthusiasm and concern. The Technology Acceptance Model framework applied in seven studies consistently demonstrated that perceived usefulness, rather than perceived ease of use, was the primary predictor of LLM adoption—a finding with direct implications for how educational technology integration programmes should be designed [22]. The faculty-student perception gap identified in this review (51% versus 74% positive) mirrors faculty-student attitudinal divergences observed during prior cycles of educational technology integration, including the introduction of simulation-based learning and e-portfolios [23].

Implications for practice and policy

Three practice-level implications of particular importance emerge from this synthesis. First, medical examination design must urgently adapt to the demonstrated LLM performance level. Assessments relying predominantly on factual recall in multiple-choice format are particularly vulnerable to LLM completion and should be supplemented or replaced with formats evaluating clinical reasoning, uncertainty management, and applied judgement—capacities where LLMs currently fall demonstrably short [23].

Second, LLM integration into medical curricula should be framed as a learnable competency rather than a prohibited tool or an unregulated resource. Competency-based medical education frameworks should formally incorporate AI literacy—defined minimally as the ability to critically appraise, verify, and appropriately weight LLM-generated content—as an explicit and assessable learning outcome [24].

Third, institution-specific academic integrity policies that distinguish constructive from dishonest AI use are urgently needed. The documented 61% academic integrity concern among students, combined with self-reported actual dishonest use of only 11%, suggests that students are capable of drawing this distinction when provided with clear frameworks. Blanket prohibition policies, in the absence of consistent enforcement capacity, are likely ineffective and risk creating institutional credibility deficits.

Strengths and limitations

This review’s principal strengths include its multi-domain scope, the application of two validated risk-of-bias tools, adherence to SWiM synthesis reporting, pre-specified subgroup analyses, and inclusion of studies from 19 countries spanning high-income and LMIC settings—providing a degree of geographic representativeness absent from prior reviews.

Seven principal limitations require transparent disclosure. First, this review was not prospectively registered in PROSPERO or an equivalent registry; while an a priori protocol was developed internally specifying eligibility criteria, search strategy, and synthesis approach, the absence of external registration means the protocol could not be publicly verified in advance, and this should be considered when weighing the review’s methodological rigour. Second, the rapid iterative evolution of LLM technology means that findings derived predominantly from studies of GPT-3.5 and early GPT-4 may not accurately represent the current performance ceiling of GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, or Llama 3 variants; the temporal validity of this review will require reassessment within 12–18 months… Third, the predominance of high-to-moderate risk of bias across included studies substantially limits confidence in causal inferences regarding LLMs’ educational effectiveness; the evidence base, while broad, is not yet methodologically mature. Fourth, publication and reporting bias is probable in this rapidly emerging field, with null or negative findings likely underrepresented, as neither PROSPERO nor clinical trial registries systematically capture planned LLM evaluation studies. Fifth, the English-language restriction may have excluded relevant evidence from Chinese-, Spanish-, Portuguese-, and German-medium medical institutions, limiting the geographic representativeness of perception and knowledge acquisition data. Sixth, the heterogeneity in perception survey instruments—validated in only 25% of studies—significantly limits psychometric comparability and precludes effect size estimation for perception outcomes. Seventh, all knowledge acquisition effect size estimates derive from short-duration interventions (≤ 8 weeks), and no included study followed students beyond a single academic term, leaving the durability of any LLM-facilitated knowledge benefit untested.

Future research priorities

The most pressing research gaps identified by this review are: (1) longitudinal cohort studies following LLM-exposed and unexposed medical student cohorts through clinical training, with clinical reasoning competency as the primary long-term outcome; (2) development and validation of standardised psychometric instruments for measuring LLM perceptions, AI literacy, and hallucination identification competency in medical education populations; (3) adequately powered randomised controlled trials or stepped-wedge designs evaluating specific LLM-based pedagogical interventions with validated endpoints; (4) equity analyses examining whether LLM access, adoption patterns, and outcomes differ across socioeconomic, gender, disability, and geographic dimensions; and (5) development of an international consensus reporting standard for LLM evaluation studies in medical education—analogous to CONSORT for RCTs—mandating reporting of exact prompts, model version, temperature settings, and hallucination assessment methodology.

Conclusions

ChatGPT, in its GPT-4 and GPT-4o iterations, has demonstrably achieved examination-passing clinical knowledge on the world’s major medical licensing assessments, but this performance describes only part of the picture. The capabilities most essential to safe clinical practice—integrating uncertain and conflicting information, calibrating clinical confidence, and interpreting multimodal data—remain outside current LLM competency at examination-grade levels. Knowledge acquisition benefits, while real, are short-lived and require structured conditions; unguided adoption carries demonstrable risks to critical appraisal development.

The most urgent message from this review is institutional rather than technological: more than four-fifths of undergraduate medical students are navigating a transformative educational technology without any formal guidance. Medical schools, accreditation bodies, and national licensing agencies must move from reactive to proactive stances—developing clear, evidence-based policies that distinguish constructive AI use from academic dishonesty, incorporate AI literacy as a measurable competency, and adapt assessment design to the demonstrated LLM performance landscape.

Future physicians will practise in environments pervaded by AI tools. The question is no longer whether LLMs belong in medical education, but how to integrate them in a manner that enhances rather than supplants the clinical reasoning development that safe medical practice demands.

Abbreviations

AI

Artificial intelligence

GPT

Generative pre-trained transformer

IQR

Interquartile range

LMIC

Low- and middle-income country

LLM

Large language model

MBBS

Bachelor of Medicine, Bachelor of Surgery

MeSH

Medical Subject Headings

MLA

Medical Licensing Assessment

NEET-PG

National Eligibility cum Entrance Test for Postgraduate

NOS

Newcastle-Ottawa Scale

NR

Not reported

OSCE

Objective Structured Clinical Examination

PLAB

Professional and Linguistic Assessments Board

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

RCT

Randomised controlled trial

ROBINS-I

Risk of Bias in Non-randomised Studies of Interventions

RQ

Research question

SWiM

Synthesis Without Meta-analysis

TAM

Technology Acceptance Model

UG

Undergraduate

USMLE

United States Medical Licensing Examination

Authors’ contributions

PRB: Conceptualisation, search strategy development, title-abstract screening, full-text screening, data extraction, risk-of-bias assessment, synthesis, writing of original draft, and arbitration of screening and data extraction discrepancies. RJ: Title-abstract screening, full-text screening, data extraction, risk-of-bias assessment, and writing (review and editing). Both authors read and approved the final manuscript.

Funding

Open access funding provided by Datta Meghe Institute of Higher Education and Research. This research received no specific funding from public, commercial, or not-for-profit sectors. No honoraria, grants, or other forms of payment were received in connection with this review.

Data availability

The full extracted dataset, complete risk-of-bias scoring tables (Additional file 3), and the complete study characteristics table for all 43 included studies (Additional file 4) are submitted with this manuscript. Complete database search strings for all databases are provided in Additional file 2. Anonymised study-level data are available from the corresponding author upon reasonable written request.

Declarations

Ethics approval and consent to participate

This study is a systematic review of previously published literature. It involved no direct human participants, no biological material, and no identifiable personal data. Formal ethics committee review is therefore not required in accordance with BMC Medical Education’s policy and international systematic review reporting standards. The primary surveys cited within included studies were reported as having received approval from respective institutional review boards, as documented in the original publications.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.