Skip to main content
Frontiers in Medicine logoLink to Frontiers in Medicine
. 2026 Jul 21;13:1888765. doi: 10.3389/fmed.2026.1888765

The temporal changes in GPT-4 performance on UKMLA practice questions: educational and clinical implications

Ravanth Baskaran 1,2,†, Sai Sirikonda 3,†, Aditya Singh 2,†, Sripradha Srinivasan 1,2, Becky Leveridge 2,4, Octavi Casals-Farre 2,5, Harmeena Kaur 6, Susruta Manivannan 1, Kabilan Elangovan 7,8, Chrystie Wan Ning Quek 7, Kanae Fukutsu 7,8,9, Judith Cave 1, Bhaskar Kumar Somani 1, Daniel Shu Wei Ting 7,8, Athanasios Hassoulas 2,*
PMCID: PMC13434848  PMID: 42553655

Abstract

Background

Generative Artificial Intelligence (GAI) models, such as GPT-4, have been extensively studied for their integration into medical practice and education. GPT-4 has demonstrated excellent performance on medical licensing examinations, including the United Kingdom Medical Licensing Assessment (UKMLA). However, the field lacks longitudinal data on whether such performance is stable or varies over time. Theoretically, iterative improvements updated by the vendor should enhance performance, but empirical evidence of such longitudinal trends remains limited. Given that GPT-4 undergoes periodic vendor-side updates, we aimed to analyse the categorical, time-spaced performance of GPT-4 on the UKMLA to characterise how its performance changes over time in a medical context.

Methods

Two publicly available UKMLA papers were fed into GPT-4 at two different time points, June 2023 and December 2024. 191 questions were provided with and without multiple-choice options to assess GPT-4’s clinical competence. McNemar’s test was performed to evaluate changes in GPT-4’s performance over time, comparing domain-specific questions.

Results

GPT-4’s accuracy improved noticeably between the two rounds (MCQ: 88.0 to 93.7%, p = 0.027; non-MCQ: 68.1 to 81.7%, p = 1.00). Single-step accuracy rose from 73.1 to 82.3%, and multi-step from 57.4 to 80.3% without MCQ. GPT-4 showed improved accuracy from Round 1 to Round 2 for both single-step and multi-step questions, with MCQ-prompted responses consistently outperforming non-MCQ responses (up to 95.1% accuracy for multi-step MCQ questions in Round 2). GPT-4’s performance improved across all question categories from round one to round two, most notably in management questions without MCQ options (+23.30%), though these differences were not statistically significant.

Discussion and conclusion

GPT-4’s performance on the UKMLA improved significantly over 18 months, suggesting that iterative vendor-side model updates enhance clinical reasoning capabilities. These findings indicate that GPT-4 may serve as a supplementary educational tool for medical students and clinicians; however, the underlying drivers of performance changes remain opaque, and such tools should be deployed with structured oversight to prevent overreliance.

Keywords: artificial intelligence, GPT-4, machine learning, medical education, UKMLA

Introduction

The rapid advancement of technology has resulted in what Alan Turing described as thinking machines, Generative Artificial Intelligence (GAI) (1). These GAI utilise neural transformer models that can process vast amounts of data, enabling the creation of complex natural language processing and coherent, constructive output (2, 3). OpenAI has not publicly disclosed the parameter count or training data volume for GPT-4; however, we know GPT-3 was trained on 175 billion parameters using approximately 570 gigabytes of publicly available online information. The scale of GPT-4’s architecture likely substantially exceeds that of GPT-3, based on observed performance capabilities and architectural innovations, though exact specifications remain proprietary (4–8).

GPT-4’s ability to sift through large volumes of backend vector datasets and provide coherent, logical, and structured responses sparks considerable excitement about AI’s role in clinical practice and medical education (9, 10). The model could be used as a virtual teaching assistant, providing instant, 24/7 feedback and reducing educators’ time and resource demands (11, 12). It could also be utilised to personalise curriculum plans tailored to an individual’s learning needs, ensuring that information is communicated and understood effectively by the end user (13, 14). In a clinical setting, GPT-4 could be used as a decision-aiding tool for clinicians, streamlining pathways and providing evidence-based suggestions (15, 16). Early studies analysing the performance of GAI in answering questions about chronic conditions, gastro-oesophageal reflux, and colorectal cancer demonstrate that GAI can also serve as a patient information source, reducing clinicians’ burden and providing an interactive resource for patients (17, 18).

However, the field lacks longitudinal data on whether such performance is stable or varies over time. Furthermore, many studies have highlighted variability in performance and reliability, including the potential to produce incorrect information or inaccurate explanations of medical concepts, a phenomenon known as hallucinations (19, 20). Despite this, further studies have shown that improvements in newer GAI models have enhanced their clinical capabilities (21, 22).

If GPT-4 is to be trusted as an information source by students, clinicians, and patients, it is vital that its performance be regularly evaluated to enable ongoing improvement and to identify changes over time. Medical licensing exams provide a broad, standardised benchmark for comparing GAI performance, and many studies have assessed GAI performance on these tests (23). Studies by Gilson et al. (24) and Kung et al. (12) demonstrated that GPT-4 exhibited knowledge and skills that surpassed those of the average learner on the United States Medical Licensing Examination (USMLE), with high levels of concordance and insight into its abilities and explanations. Similarly, GPT-4 has exceeded the minimum competency threshold, as demonstrated by the German medical board (25). Furthermore, studies by Casals-Farre et al. (26) and Lai et al. (27) revealed that GPT-4 performs exceptionally well on the United Kingdom Medical Licensing Assessment (UKMLA), passing both exams with high scores.

Theoretically, with the exponential increase in the amount of data on the internet feeding into LLMs’ datasets, natural language processing models such as GPT-4 should be able to adapt and enhance their knowledge (28–30). In this study, we evaluated GPT-4’s temporal performance on the UKMLA through a categorical analysis of questions and their curated answers to assess improvements in the model over an 18-month period.

Methodology

Question source

Sample questions were obtained from the Medical School Council (MSC) website (31). The website was first accessed on June 10th, 2023, and then again on December 16th, 2024. The first iteration (2023) of mock questions available on MSC’s website was used throughout this study.

Twenty-four areas of clinical practice are tested across two papers in the UKMLA Applied Knowledge Test (AKT) (28). Each paper consists of 100 questions, including nine image-based questions in the 2023 iteration. Each question includes a question stem and five multiple-choice options. In our study, 191 questions were included because GPT-4 did not process images when its performance was first assessed in 2023. To standardise our data, we excluded these image-based questions from both analyses.

Model and testing environment

Questions were administered via the consumer ChatGPT web interface1 using a ChatGPT Plus subscription, with the model selector set to “GPT-4”. As the consumer interface does not expose sampling parameters, temperature, top-p and maximum token settings were left at their platform defaults and could not be modified. No system prompt or custom instructions were configured. The memory feature was disabled, and no plugins, tools, file uploads, or advanced data analysis were used; web browsing was enabled. Each question was entered into a new, fresh chat session with no prior conversation history, and the chat was closed before the next item to prevent context carryover. Each question was submitted once (a single attempt) (Table 1).

Table 1.

Model configuration and testing environment parameters.

Parameter Round 1
Testing dates Round 1: 10–15 June 2023; Round 2: 16–19 December 2024
Exact model GPT-4 (both rounds)
Access route Consumer ChatGPT web interface (chat.openai.com)
Model selector setting GPT-4
Web browsing On
Plugins/tools None
File uploads/Advanced data analysis Not used
Memory feature Disabled
System prompt/custom instructions None (left blank)
Temperature Platform default; not user-modifiable
Top-p/maximum tokens Platform default; not user-modifiable
Prompt wording – non-MCQ Question stem only (clinical scenario + leading questions)
Prompt wording – MCQ Question stem + all five answer options
Attempts per questions 1
Conversation reset New, fresh chat per input; chat closed before next input

Large language model (LLM) inputs

Each paper was evaluated separately. The question stem, including the scenario and leading question, was first input into GPT-4 without answer options, and this response was then recorded under the “non-MCQ” section. The entire question, along with five MCQ prompts, was subsequently entered into a separate, fresh chat, and the response was recorded under the “MCQ prompt” section. The identical GPT-4 configuration described above was used on both occasions, enabling standardisation. The model was not primed with any preliminary information on either occasion to recreate the typical day-to-day use of a medical student/clinician.

Data collection

The data were collected between 10/06/2023 and 15/06/2023 (Round 1) and again between 16/12/2024 and 19/12/2024 (Round 2) using the same methods at both time points, enabling longitudinal comparison across an approximate 18-month interval.

The accuracy of answers provided by the GPT-4 was coded as per the following categories:

  1. Correct: the answer provided is entirely correct and correlates with the marking scheme.

  2. Incorrect: the answer provided is entirely inaccurate and does not adhere to the marking scheme.

  3. Indeterminate: the answer provided is neither entirely accurate nor entirely inaccurate.

One author, a medical doctor (SS), independently determined the answer category, and any queries were assessed separately by a senior author. These were determined at the beginning of the study, during the first iteration of data collection, using the same categories used subsequently.

Non-MCQ condition

The non-MCQ condition presented the question stem alone, without answer options. This format represents a free-response approximation of the UKMLA task, which is designed as a single-best-answer MCQ. Free-text responses were evaluated against the official mark scheme using the three-category rubric above. A response was classified as correct if it identified the correct answer, whether stated alone or embedded within additional reasoning or detail. A response was classified as indeterminate if it demonstrated partial understanding or provided a clinically reasonable answer that did not identify the correct answer. All other responses were classified as incorrect. Supplementary Table 1 provides examples of our scoring rubric.

Question categorisation

Questions were initially categorised by complexity based on the number of logical steps needed to solve them. A question was classified as either single-step or multistep. A question was classified as either a single-step question (requiring one logical deduction) or a multistep question (requiring two or more stages of analysis). For example, a multistep question might require diagnosing a condition based on the clinical presentation and then selecting a second-line treatment due to contraindications to the first medication. One author (SS), a medical doctor, independently determined the complexity category in a blinded manner. Any queries were resolved by the senior author. These were determined at the beginning of the study, during the first iteration of data collection, using the same categories used subsequently.

GPT-4’s domain-specific strengths were then analysed by assessing its performance in different competencies. We categorised questions into five broad competencies as follows:

  1. Diagnosis: identifying the underlying cause of clinical presentation.

  2. Investigation: choosing the most appropriate test from bedside lab tests to imaging within the context of the clinical vignette.

  3. Core medical knowledge: applying essential principles of anatomy, physiology, and biochemistry.

  4. Management: choosing the most suitable treatment plan within the context of the clinical vignette.

  5. Pharmacology/Prescribing: understanding drug actions, adverse reactions, contraindications, and therapeutic prescribing.

Each question was independently categorised into one of the competency domains by a medical doctor (SS). Queries were resolved by a senior author. Because some questions are frequently mapped to multiple competency domains, we applied a hierarchical rule: the competency domain deemed most relevant and critical to selecting the correct answer. This approach ensured mutually exclusive categorisation, enabling unambiguous domain-specific analysis. Categories were determined at the beginning of the study, during the first iteration of data collection, and were used subsequently.

Overall question accuracy was observed both with and without MCQ prompts. These were then analysed in terms of question complexity and assessed competency. All results were analysed using IBM SPSS version 29, with descriptive statistics and McNemar’s test.

Results

Our primary hypothesis predicted improved overall accuracy in GPT-4’s performance when attempting the UKMLA mock during the second round of data collection.

Overall performance

During the initial data collection phase (June 2023), GPT-4 achieved a total score of 88.0% (168/191) across both papers one and two with MCQ options. This score dropped to 68.1% (130/191) when MCQ options were not provided. In the second phase (December 2024), GPT-4’s overall performance improved to 93.7% (179/191) for MCQs and 81.7% (156/191) for the non-MCQ set. A McNemar test comparing the correlation between “non-MCQ” and MCQ performances across both time points. For crosstabulation during the McNemar test, all indeterminate responses were counted as incorrect. Hence, the two categories used were “correct” and “incorrect”. Comparing “non-MCQ” responses across both time points, the test showed a p-value of 1.00, indicating statistical non-significance. Comparing overall performance in the “MCQ” sections between the first and second phases showed a statistically significant difference (p = 0.027).

No indeterminate questions were answered on either occasion when MCQ prompts were provided. The level of indeterminacy decreased between the two data stages when no MCQs were provided, with 19 indeterminate answers in the first stage and 12 in the latter.

Step analysis

We calculated a total of 130 single-step and 61 multi-step questions across both papers. During the first round, GPT-4 answered 95/130 (73.1%) single-step and 35/61 (57.4%) multi-step questions without MCQ options. This improved to 117/130 (90.0%) and 51/61 (83.6%) correct single- and multi-step questions with MCQ prompts, respectively. During the second round, GPT-4 answered 107/130 (82.3%) single-step and 49/61 (80.3%) multi-step questions without MCQs; and 121/130 (93.1%) single-step and 58/61 (95.1%) multi-step questions with MCQs (Figures 1, 2).

Figure 1.

Bar chart compares percentage of correct answers for single-step and multi-step questions, with results from Round 1 (purple) and Round 2 (teal). Both categories show a higher percentage of correct answers in Round 2.

GPT-4’s performance in single-step and multistep questions with the MCQ options provided.

Figure 2.

Bar chart comparing percentage of correct answers in two groups: single step (n equals 130) and multi step (n equals 61), across two rounds. Both groups show higher correct answer percentages in round 2 (teal bars) compared to round 1 (purple bars). Round 2 multi step and single step performance are both above 75 percent, while round 1 bars are lower, with single step above fifty percent and multi step around fifty percent. MCQ and NMCQ question types are referenced in the legend.

GPT-4’s performance in single-step and multistep questions without the MCQ options provided.

Competence analysis

There were 57 diagnoses, 27 investigations, 32 core medical knowledge, 43 management, and 32 pharmacology questions.

Management questions showed the greatest differences between the two rounds, without the MCQ option. In diagnosis questions, when provided with MCQ prompts, GPT-4 performed 5.30% better and 7.02% better without MCQ options in the second round. The most notable improvement was seen in “management” questions without MCQ, where GPT-4’s performance increased by 23.30% in round two. The least improvement occurred in “management” and “core medical knowledge” questions with MCQ options, with only a 2.30 and 3.10% improvement, respectively.

GPT-4 scored 100% in round two of “investigation” questions when MCQs were provided. The lowest score was in round one “management” questions, where no MCQ prompts were given, with GPT-4 scoring 51.20%.

Figure 3 illustrates the competence-specific performance trends across both prompting rounds. The “MCQ Round 2” line shows the highest accuracy across all competencies, ranging from 88 to 100%. This is followed by “MCQ Round 1” (92–92%), “Non-MCQ Round 2” (74–90%) and finally “Non-MCQ Round 1” (50–83%). The most pronounced gain in accuracy is seen in “management” questions under non-MCQ conditions, where a marked trough from round 1 (51.2%) shifts upwards to 74.5% in round 2, aligning with the reported 23.3% improvement. GPT-4 shows high accuracy in diagnosis and pharmacology in round 1 and continues to increase in round 2. Core medical knowledge and management questions show the smallest gains in MCQ-related performance between the two rounds. Overall, the curves illustrate the largest improvements in free-response answers (non-MCQ) between the two rounds.

Figure 3.

Line graph compares percentage of correct answers across competence levels one to five for four formats: MCQ Round 1, NMCQ Round 1, MCQ Round 2, NMCQ Round 2. MCQ lines remain higher throughout.

GPT-4’s performance across all five competencies in rounds one and two with and without MCQ options.

Statistically significant results based on McNemar testing are highlighted in Table 2, with N = 191 and α = 0.05 for all values (Tables 2, 3).

Table 2.

The overall performance of GPT-4 through both rounds of data collection.

ROUND 1 ROUND 2
Without-MCQ options With MCQ options Without-MCQ options With MCQ options
Correct 130 168 156 179
Incorrect 42 23 23 12
Indeterminate 19 0 12 0

Table 3.

GPT-4’s performance on each competency in each round, along with the overall change between the two rounds, as assessed using the McNemar test.

With MCQ Without MCQ
Round 1(R1) Round 2 (R2) △1 (R2-R1) p-value Round 1(R3) Round 2 (R4) △2 (R4-R3) p-value
Diagnosis (n = 57) 91.20% 96.50% 5.30% 0.25 84.20% 91.20% 7.00% 0.125
Investigation (n = 27) 92.60% 100.00% 7.40% - 63.00% 70.40% 7.40% 0.687
Knowledge (n = 32) 87.50% 90.60% 3.10% 1.00 71.90% 84.40% 12.50% 0.344
Management (n = 43) 86.05% 88.40% 2.35% 1.00 51.10% 74.40% 23.3% 0.052
Pharmacology (n = 32) 81.30% 93.80% 12.50% 0.125 62.50% 81.30% 18.80% 0.146

The p-value for the comparison between the two rounds is also provided for each instance.

Discussion

Over an 18-month longitudinal evaluation, GPT-4 demonstrated significantly improved performance in answering UKMLA questions. This represents a significant improvement and suggests that the deployed GPT-4 model has become more capable at medical reasoning and clinical decision-making as assessed by this high-stakes examination standard.

However, GPT-4’s improvement cannot be attributed solely to the model’s autonomous learning. The consumer GPT interface does not provide access to dated model snapshots, and OpenAI deploys updates to the underlying GPT-4 model without public disclosure. The observed performance change, therefore, may reflect vendor-side model updates, changes in training data composition, modifications to alignment protocols, stochastic variation, or a combination of these factors. We cannot distinguish between these mechanisms based on available evidence.

Our findings are consistent with documented updates to GPT-4’s training and alignment pipeline, including iterative refinements to (1) the training datasets, (2) the underlying neural network parameters, and (3) the natural language understanding and generation mechanisms. GPT-4 has been shown to undergo iterative refinements involving retraining on updated datasets and reinforcement learning from human feedback (RLHF) (29). RLHF enables continual behavioral improvement by structuring human feedback from both expert labellers and real-world user interactions. Prior work has demonstrated that large language models can achieve substantial gains on complex reasoning tasks with chain-of-thought prompting, in which intermediate reasoning steps are elicited or internally modelled to support more accurate final answers (30). While such reasoning traces may not be explicitly exposed to end users, their incorporation into prompting strategies and model optimisation frameworks plausibly contributes to improved medical reasoning performance over successive model iterations. The primary driver of longitudinal performance change in GPT-4 is likely backend model and alignment modifications across releases, rather than autonomous learning from individual user interactions (32). Our observed improvement is therefore consistent with structured, vendor-managed updates to training and alignment pipelines.

There was a statistically significant improvement in GPT-4’s performance on questions with multiple-choice options compared to those with open-ended options. Studies have shown that multiple-choice questions (MCQs) support rote memorisation, whereas open-ended questions (OEQs) favour conceptual learning (33, 34). Answering MCQs requires reliance on pattern recognition and familiarity-based answer identification, whereas OEQs require active engagement, understanding, and the ability to derive information from a data source. Since GPT-4 demonstrates a significant increase in UKMLA questions when presented as MCQs, this indicates that its familiarity-based understanding has improved (35, 36). More needs to be done to prime and enable GPT-4 to better understand these concepts, so that knowledge is application-based rather than recognition-based.

Domain-specific analysis showed no notable increases across domains, which remained generally unchanged. As an inanimate entity, GPT-4 relies on building from its foundational models to improve its capabilities (37). Better performance results from further training and human feedback (38). This suggests that ongoing training and teaching of GPT-4 to foster the overall development of its abilities will lead to better clinical correlations and applications.

The pronounced trough and subsequent rise in management question accuracy under non-MCQ conditions reflect the higher cognitive burden of multi-step synthesis involved in clinical management. This could possibly be less well captured by pattern-recognition-based formats used by LLMs. In contrast, GPT-4 demonstrates near-ceiling performance on investigative questions using MCQ prompts. Across diagnoses, core medical knowledge and pharmacology, non-MCQ scores moved closer to MCQ scores, suggesting better recall and application without prompts in round two. By contrast, a persistent gap between MCQ and non-MCQ scores in management questions indicates that GPT-4’s improved clinical reasoning does not translate uniformly into all domains. This aligns with the literature, which highlights the importance of contrasting competing options as the foundation of clinical management, rather than relying solely on recall of facts (19, 39–41).

We used the publicly available UKMLA question paper published by the Medical Schools Council (MSC) on their official website. Two question papers with worked answers are accessible on the MSC site. Although it is challenging to identify the specific datasets used to train GPT-4, the availability of accurate answers to the questions we input should enable it to achieve a perfect 100% score on both papers at both time points, since we allowed web browsing. However, since this is not observed, hypotheses can be formulated to explain GPT-4’s performance in the UKMLA.

AI hallucination is a widely discussed concept within the AI scholarly community, which may help explain why GPT-4 has not achieved perfect scores on examinations. It is defined as the generation of fictional information based on associations formed around the most probable occurrence (42). Some critics of GAI have suggested that GPT-4 identifies a string of words that most closely relates to the question and coherently expresses it without any form of thinking or reasoning (43). Hallucinations are often classified into three categories: input, fact, and context, which can lead to the production of serious and incorrect content for the end user (38). Since medicine is evidence-based, a wide range of diverse information is available for nearly every medical condition, diagnosis, and treatment option (44). While it is the responsibility of the physician to understand the levels of scientific evidence and study strengths, and to incorporate this into their practice, the neural networks of GPT-4 may lack these capabilities, potentially explaining the gap between the expected and actual scores e of the GAI in the UKMLA (39, 45).

Mirroring is a concept rarely discussed in the literature, but it could help explain the gap between GPT-4’s theoretical score and its actual performance in the UKMLA. In GAI, mirroring refers to the way it perpetuates pre-existing human biases. Since GAIs are trained on datasets mainly sourced from humans, they naturally embody systemic conscious and unconscious biases and understandings that humans also experience (40). While these biases could lead to misinformation if amplified, they may also account for the differences seen in UKMLA results. Because human nature involves errors, developing a human-like LLM such as GPT-4 will inherently lead to human-like fallacies (19, 41, 46).

The results of this study indicate that GPT-4’s performance on the UKMLA question bank improved substantially over an 18-month period, reflecting iterative vendor-side updates to training data, alignment strategies, and model capabilities. While the consumer interface label remains “GPT-4”, the underlying model has likely evolved through backend refinements, as evidenced by the observed performance gains (47).

GPT-4 has shown significant enhancements in its capabilities within the UKMLA, with improvements across multiple specific domains. These findings support the inclusion of GPT-4 as a supplementary tool in medical education, where it could provide formatted explanations of clinical concepts or help identify knowledge gaps to inform personalised learning (48, 49). The significant improvements also suggest that GPT-4 has the potential to be utilised by clinicians as a decision-aiding tool or a source of patient information. However, performance evaluations of GAI triaging in real-world emergency scenarios have yielded heterogeneous and limited results. Hence, further training and evaluation of models in real-life scenarios are required before real-world implementation (50). Medical educators and clinicians considering GPT-4 integration should treat these findings as provisional support for pilot programmes with structured evaluation, while recognising clinical implementation requires substantially more evidence.

The development of retrieval-augmented generation (RAG) offers promise for improving both medical education and the potential clinical applications of GAI (51, 52). RAG provides a cost-effective method to optimise LLM outputs by enabling it to reference an external knowledge base before generating a response. Early studies have shown that RAG enables models to deliver significantly better, more consistent responses with fewer hallucinations. Supplying GAI with medical textbooks, institutional lecture notes, or exam-oriented datasets could give students a tailored, interactive chatbot to support their learning (53, 54). Likewise, providing national guidelines, peer-reviewed literature, or specific clinical pathways from national bodies or National Health Service (NHS) trusts could improve chatbot responses for clinicians and patients, making the chatbot a reliable reference tool. Based on our findings regarding GPT-4’s capabilities in this study, RAG pairings with GPT-4 could be used to deliver accurate, targeted information to users and, eventually, to develop customised learning plans based on individual learner characteristics to enhance students’ abilities.

While our study shows the rapidly evolving and improving capabilities of GAI, such as GPT-4, we should approach their integration into clinical and medical education with cautious optimism. Potential risks of continuing to use GPT-4 in medical training and clinical settings include: (1) a lack of understanding and interpretation of the information processed and generated, (2) the absence of ethical principles governing the use of GAI, (3) the spread of misinformation, and (4) an over-reliance on LLMs by students and medical professionals, which could weaken critical and lateral thinking skills. Therefore, resources such as GPT-4 should be used carefully, with users verifying the information provided (38, 55).

This study has several limitations. First, evaluations were conducted using the standard GPT-4 user interface without applying temperature optimisation, system-level prompt controls, or advanced prompt engineering techniques. While these strategies may improve model performance, they were intentionally excluded to reflect the typical usage patterns of lay clinicians and medical trainees, for whom such optimisations are not routinely accessible. Second, the analysis focuses on observable performance and knowledge drift over time rather than isolating specific backend mechanisms responsible for these changes, as model training details and update schedules remain opaque. Thirdly, results may not generalise to API-based deployments or alternative prompting configurations, where greater control over inference parameters and model snapshots is available. Fourthly, web browsing was enabled during testing; as the MSC mock papers are freely accessible online, the model may have retrieved question or answer content from the web rather than relying solely on internal clinical knowledge, potentially inflating accuracy. Finally, each question was administered only once, so within-model response variability was not captured (56).

Conclusion

GPT-4’s performance in medical licensing exams shows promise in producing clinically capable GAI. Our longitudinal study of GPT-4’s results on the same UKMLA paper highlights notable improvements across categories, demonstrating the extensive developmental potential of these GAI in medicine. Institutions should consider using AI LLMs as educational aids to create learner-specific programmes that assist medical students. Any clinical application, such as a decision-support adjunct or patient information resource, should be considered provisional and would require substantially more validation before wider clinical deployment. However, caution is necessary when relying on these GAI, due to concerns such as (1) the dissemination of misinformation, (2) the teaching of incorrect concepts, and (3) the risk of creating de-skilled physicians through excessive dependence on GAI in clinical practice.

Acknowledgments

We would like to thank MedTechEazy, OSCEazy and Cardiff University Hybrid Interactive Virtual Environments (HIVE) Digital and Teaching Innovation Unit for their support. Members of these organisations conceptualised and wrote this paper. These organisations did not provide any commercial, educational, or institutional support, and no conflicts of interest were identified among their members.

Funding Statement

The author(s) declared that financial support was not received for this work and/or its publication.

Edited by: Cristian Vacacela Gomez, Universidad Tecnológica Ecotec, Ecuador

Reviewed by: Nilesh Kumar Mitra, International Medical University, Malaysia

Julian Madrid, Ortenau Klinikum, Germany

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

RB: Project administration, Resources, Writing – original draft, Formal analysis, Validation, Visualization, Data curation, Conceptualization, Writing – review & editing, Methodology, Investigation, Software. SaS: Project administration, Methodology, Writing – review & editing, Formal analysis, Writing – original draft, Data curation. AS: Resources, Validation, Formal analysis, Project administration, Writing – original draft, Visualization, Investigation, Data curation, Writing – review & editing, Conceptualization, Methodology, Software. SrS: Writing – review & editing, Project administration, Methodology, Data curation. BL: Data curation, Methodology, Writing – review & editing. OC-F: Writing – review & editing, Conceptualization, Project administration, Validation. HK: Writing – review & editing, Methodology, Project administration, Data curation. SM: Validation, Project administration, Supervision, Methodology, Writing – review & editing. KE: Methodology, Formal analysis, Project administration, Data curation, Supervision, Validation, Writing – original draft, Writing – review & editing. CQ: Project administration, Methodology, Visualization, Writing – review & editing, Resources. KF: Visualization, Validation, Writing – review & editing, Supervision. JC: Supervision, Visualization, Validation, Writing – review & editing. BS: Supervision, Writing – review & editing, Visualization, Validation. DT: Visualization, Validation, Writing – review & editing, Supervision. AH: Validation, Writing – review & editing, Supervision, Visualization.

Conflict of interest

The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author DT declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. Generative Artificial Intelligence was used in the manuscript's methods due to the nature of the study.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2026.1888765/full#supplementary-material

Table_1.XLSX (31.2KB, XLSX)
Table_2.DOCX (16.3KB, DOCX)

References

  • 1.Turing AM. Computing machinery and intelligence. Mind. (1950) LIX:433–60. doi: 10.1093/mind/LIX.236.433, 42393199 [DOI] [Google Scholar]
  • 2.Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., et al. Attention is All you Need Advances in Neural Information Processing Systems Curran Associates, Inc. (2017). (Accessed Oct 14, 2025). Available online at: https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  • 3.Madani A, Krause B, Greene ER, Subramanian S, Mohr BP, Holton JM, et al. Large language models generate functional protein sequences across diverse families. Nat Biotechnol. (2023) 41:1099–106. doi: 10.1038/s41587-022-01618-2, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Li C. Openai’s GPT-3 Language Model: A Technical Overview (2020) (cited 2026 June 27). Available online at: https://lambda.ai/blog/demystifying-gpt-3
  • 5.Brown T., Mann B., Ryder N., Subbiah M., Kaplan J. D., Dhariwal P., et al. Language Models are Few-Shot Learners Advances in Neural Information Processing Systems Curran Associates, Inc. (2020) (Accessed Oct 14, 2025). p. 1877 1901. Available online at: https://papers.nips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  • 6.Teubner T, Flath CM, Weinhardt C, van der Aalst W, Hinz O. Welcome to the era of ChatGPT et al. Bus Inf Syst Eng. (2023) 65:95–101. doi: 10.1007/s12599-023-00795-x [DOI] [Google Scholar]
  • 7.OpenAI. Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, et al. GPT-4 technical report. arXiv. (2024). doi: 10.48550/arXiv.2303.08774 [DOI] [Google Scholar]
  • 8.Hoffmann J, Borgeaud S, Mensch A, Buchatskaya E, Cai T, Rutherford E, et al. Training compute-optimal large language models. arXiv. (2022). doi: 10.48550/arXiv.2203.15556 [DOI] [Google Scholar]
  • 9.Ahmad Z, Rahim S, Zubair M, Abdul-Ghafar J. Artificial intelligence (AI) in medicine, current applications and future role with special emphasis on its potential and promise in pathology: present and future impact, obstacles including costs and acceptance among pathologists, practical and philosophical considerations. A comprehensive review. Diagn Pathol. (2021) 16:24. doi: 10.1186/s13000-021-01085-4, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Tozsin A, Ucmak H, Soyturk S, Aydin A, Gozen AS, Fahim MA, et al. The role of artificial intelligence in medical education: a systematic review. Surg Innov. (2024) 31:415–23. doi: 10.1177/15533506241248239, [DOI] [PubMed] [Google Scholar]
  • 11.Chan KS, Zary N. Applications and challenges of implementing artificial intelligence in medical education: integrative review. JMIR Med Educ. (2019) 5:e13930. doi: 10.2196/13930, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Kung TH, Cheatham M, Medenilla A, Sillos C, Leon LD, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. (2023) 2:e0000198. doi: 10.1371/journal.pdig.0000198, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Lambert J, Stevens M. ChatGPT and generative AI technology: a mixed bag of concerns and new opportunities. Comput Sch. (2024) 41:559–83. doi: 10.1080/07380569.2023.2256710 [DOI] [Google Scholar]
  • 14.Denny JC, Spickard A, Speltz PJ, Porier R, Rosenstiel DE, Powers JS. Using natural language processing to provide personalized learning opportunities from trainee clinical notes. J Biomed Inform. (2015) 56:292–9. doi: 10.1016/j.jbi.2015.06.004, [DOI] [PubMed] [Google Scholar]
  • 15.Huang J, Neill L, Wittbrodt M, Melnick D, Klug M, Thompson M, et al. Generative artificial intelligence for chest radiograph interpretation in the emergency department. JAMA Netw Open. (2023) 6:e2336100. doi: 10.1001/jamanetworkopen.2023.36100 37796505, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Ferdush J, Begum M, Hossain ST. ChatGPT and clinical decision support: scope, application, and limitations. Ann Biomed Eng. (2024) 52:1119–24. doi: 10.1007/s10439-023-03329-4, [DOI] [PubMed] [Google Scholar]
  • 17.Fahad N, Rabbi RI, Benta Hasan S, Sultana Prity F, Ahmed R, Ahmed F, et al. Generative AI in clinical (2020–2025): a mini-review of applications, emerging trends, and clinical challenges. Front Digit Health. (2025) 7:1653369. doi: 10.3389/fdgth.2025.1653369, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Schachner T, Keller R, von Wangenheim F. Artificial intelligence-based conversational agents for chronic conditions: systematic literature review. J Med Internet Res. (2020) 22:e20701. doi: 10.2196/20701, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Sun Y, Sheng D, Zhou Z, Wu Y. AI hallucination: towards a comprehensive classification of distorted information in artificial intelligence-generated content. Humanit Soc Sci Commun. (2024) 11:1278. doi: 10.1057/s41599-024-03811-x [DOI] [Google Scholar]
  • 20.Xu X, Chen Y, Miao J. Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review. J Educ Eval Health Prof. (2024) 21:6. doi: 10.3352/jeehp.2024.21.6, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Sanli AN, Turan B, Tekcan Sanli DE. Advances in large language model performance: a comparative study of ChatGPT-4 and ChatGPT-5 on ABSITE questions. Am Surg. (2025):00031348251390958. doi: 10.1177/00031348251390958, [DOI] [PubMed] [Google Scholar]
  • 22.Seth I, Marcaccini G, Lim B, Novo J, Bacchi S, Cuomo R, et al. The temporal evolution of large language model performance: a comparative analysis of past and current outputs in scientific and medical research. Informatics (Basel). (2025) 12:86. doi: 10.3390/informatics12030086 [DOI] [Google Scholar]
  • 23.Liu M, Okuhara T, Chang X, Okada H, Kiuchi T. Performance of ChatGPT in medical licensing examinations in countries worldwide: a systematic review and meta-analysis protocol. PLoS One. (2024) 19:e0312771. doi: 10.1371/journal.pone.0312771, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. (2023) 9:e45312. doi: 10.2196/45312, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Madrid J, Diehl P, Selig M, Rolauffs B, Hans FP, Busch HJ, et al. Performance of plug-in augmented ChatGPT and its ability to quantify uncertainty: simulation study on the German medical board examination. JMIR Med Educ. (2025) 11:e58375. doi: 10.2196/58375, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Casals-Farre O, Baskaran R, Singh A, Kaur H, Ul Hoque T, de Almeida A, et al. Assessing ChatGPT 4.0’s capabilities in the United Kingdom medical licensing examination (UKMLA): a robust categorical analysis. Sci Rep. (2025) 15:13031. doi: 10.1038/s41598-025-97327-2, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Lai UH, Wu KS, Hsu TY, Kan JKC. Evaluating the performance of ChatGPT-4 on the United Kingdom medical licensing assessment. Front Med. (2023) 10:10. doi: 10.3389/fmed.2023.1240915, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.GMC . Medical Licensing Assessment. (Accessed Oct 15, 2025). Available online at: https://www.gmc-uk.org/education/medical-licensing-assessment
  • 29.Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al. Training language models to follow instructions with human feedback. arXiv. (2022). doi: 10.48550/arXiv.2203.02155 [DOI] [Google Scholar]
  • 30.Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv. (2023). doi: 10.48550/arXiv.2201.11903 [DOI] [Google Scholar]
  • 31.Medical Schools Council . Practice exam for the MS AKT. (Accessed Oct 15, 2025). Available online at: https://www.medschools.ac.uk/for-students/medical-licensing-assessment/practice-exam-for-the-ms-akt/
  • 32.Chen L, Zaharia M, Zou J. How is ChatGPT’s behavior changing over time? arXiv. (2023). doi: 10.48550/arXiv.2307.09009 [DOI] [Google Scholar]
  • 33.Newble DI, Baxter A, Elmslie RG. A comparison of multiple-choice tests and free-response tests in examinations of clinical competence. Med Educ. (1979) 13:263–8. doi: 10.1111/j.1365-2923.1979.tb01511.x, [DOI] [PubMed] [Google Scholar]
  • 34.Martinez ME. Cognition and the question of test item format. Educ Psychol. (1999) 34:207–18. doi: 10.1207/s15326985ep3404_2 [DOI] [Google Scholar]
  • 35.Melovitz Vasan CA, DeFouw DO, Holland BK, Vasan NS. Analysis of testing with multiple choice versus open-ended questions: outcome-based observations in an anatomy course. Anat Sci Educ. (2018) 11:254–61. doi: 10.1002/ase.1739, [DOI] [PubMed] [Google Scholar]
  • 36.Ozuru Y, Briner S, Kurby CA, McNamara DS. Comparing comprehension measured by multiple-choice and open-ended questions. Canad J Exp Psychol. (2013) 67:215–27. doi: 10.1037/a0032918, [DOI] [PubMed] [Google Scholar]
  • 37.Adamopoulou E, Moussiades L. "An overview of Chatbot technology". In: Maglogiannis I, Iliadis L, Pimenidis E, editors. Artificial Intelligence Applications and Innovations. Cham: Springer International Publishing; (2020). p. 373–83. [Google Scholar]
  • 38.Schukow C, Smith SC, Landgrebe E, Parasuraman S, Folaranmi OO, Paner GP, et al. Application of ChatGPT in routine diagnostic pathology: promises, pitfalls, and potential future directions. Adv Anat Pathol. (2024) 31:15–21. doi: 10.1097/PAP.0000000000000406, [DOI] [PubMed] [Google Scholar]
  • 39.McAlister FA, Graham I, Karr GW, Laupacis A. Evidence-based medicine and the practicing clinician. J Gen Intern Med. (1999) 14:236–42. doi: 10.1046/j.1525-1497.1999.00323.x, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Vallor S, The AI. Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking. Oxford, United Kingdom: Oxford University Press; (2024). p. 273. [Google Scholar]
  • 41.Moore PV. The mirror for (artificial) intelligence: in whose reflection? Comp Lab L Pol J. (2019) 41:47–67. doi: 10.2139/ssrn.3423704 [DOI] [Google Scholar]
  • 42.Kumar M, Mani UA, Tripathi P, Saalim M, Roy S, Kumar M. Artificial hallucinations by Google bard: think before you leap. Cureus. (2023) 15. doi: 10.7759/cureus.43313 [Epubh ahead of print]., [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Anastasiei ID, Georgescu MR. Automated vs manual content analysis – a retrospective look. Sci Ann Econ Bus. (2020) 67:57–67. doi: 10.47743/saeb-2020-0025, 37530755 [DOI] [Google Scholar]
  • 44.Hindorff LA, Bonham VL, Ohno-Machado L. Enhancing diversity to reduce health information disparities and build an evidence base foR genomic medicine. Pers Med. (2018) 15:403–12. doi: 10.2217/pme-2018-0037, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Shahar E. A Popperian perspective of the term ‘evidence-based medicine’. J Eval Clin Pract. (1997) 3:109–16. doi: 10.1046/j.1365-2753.1997.00092.x, [DOI] [PubMed] [Google Scholar]
  • 46.Bar-Cohen Y, Hanson D, Marom A. "Mirroring Humans". In: Hanson D, Bar-Cohen Y, editors. The Coming Robot Revolution: Expectations and Fears about Emerging Intelligent, Humanlike Machines. New York, NY: Springer; (2009). p. 99–116. [Google Scholar]
  • 47.OpenAI API . GPT-4 Model. (Accessed 2025 Dec 18). Available online at: https://platform.openai.com
  • 48.Ning Y, Ong JCL, Cheng H, Wang H, Ting DSW, Tham YC, et al. How can artificial intelligence transform the training of medical students and physicians? Lancet Digit Health. (2025) 7:100900: 100900. doi: 10.1016/j.landig.2025.100900, [DOI] [PubMed] [Google Scholar]
  • 49.Zhou C, Neubert M, Koide Y, Zhang Y, Vuong VQ, Schlöder T, et al. PAL – parallel active learning for machine-learned potentials. Digital Disc. (2025) 4:1901–11. doi: 10.1039/D5DD00073D, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Benning L, Hirsch A, Gröschel M, Röschl T, Spott M, Hans FP, et al. Performance evaluation and benchmarking across 16 large language models on a comprehensive real-world emergency department triage data set. medRxiv. (2026). doi: 10.64898/2026.05.28.26353935 [DOI] [Google Scholar]
  • 51.Ozmen BB, Mathur P. Evidence-based artificial intelligence: implementing retrieval-augmented generation models to enhance clinical decision support in plastic surgery. J Plast Reconstr Aesthet Surg. (2025) 104:414–6. doi: 10.1016/j.bjps.2025.03.053, [DOI] [PubMed] [Google Scholar]
  • 52.Gao Y, Xiong Y, Gao X, Jia K, Pan J, Bi Y, et al. Retrieval-augmented generation for large language models: a survey. arXiv. (2024). doi: 10.48550/arXiv.2312.10997 [DOI] [Google Scholar]
  • 53.Liu S, McCoy AB, Wright A. Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. J Am Med Inform Assoc. (2025) 32:605–15. doi: 10.1093/jamia/ocaf008, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Ke YH, Jin L, Elangovan K, Abdullah HR, Liu N, Sia ATH, et al. Retrieval augmented generation for large language models and its generalizability in assessing medical fitness. NPJ Digit Med. (2025) 8:187. doi: 10.1038/s41746-025-01519-z, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Nakagawa K, Moukheiber L, Celi LA, Patel M, Mahmood F, Gondim D, et al. AI in pathology: what could possibly go wrong? Semin Diagn Pathol. (2023) 40:100–8. doi: 10.1053/j.semdp.2023.02.006, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Kochanek M, Cichecki I, Kaszyca O, Szydło D, Madej M, Jędrzejewski D, et al. Improving training dataset balance with ChatGPT prompt engineering. Electronics (Basel). (2024) 13:2255. doi: 10.3390/electronics13122255 [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Table_1.XLSX (31.2KB, XLSX)
Table_2.DOCX (16.3KB, DOCX)

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.


Articles from Frontiers in Medicine are provided here courtesy of Frontiers Media SA

RESOURCES