Abstract
Purpose
We examined the performance of artificial intelligence chatbots on the PREview Practice Exam, an online situational judgment test for professionalism and ethics.
Methods
We used validated methodologies to calculate scores and descriptive statistics, χ2 tests, and Fisher’s exact tests to compare scores by model and competency.
Results
GPT-3.5 and GPT-4 scored 6/9 (76th percentile) and 7/9 (92nd percentile), respectively, higher than medical school applicant averages of 5/9 (56th percentile). Both models answered 95 + % of questions correctly.
Conclusions
Chatbots outperformed the average applicant on PREview, suggesting their potential for healthcare training and decision-making and highlighting risks of online assessment delivery.
Keywords: AAMC, PREview, OpenAI, ChatGPT
Background
OpenAI’s ChatGPT is capable of human-like communication and reasoning [1]. Although its GPT-3.5 and GPT-4 models can both achieve passing scores on tests of clinical knowledge and cognitive ability such as the US Medical Licensing Examination Step exams [1], their ethical and professional decision-making capabilities are unknown.
In 2022, the Association of American Medical Colleges (AAMC) designed PREview [2], an online situational judgment test (SJT), to evaluate pre-professional competencies of medical school applicants. Although SJTs have been validated as superior predictors of clinical and professional performance than tests of cognitive ability, academic performance, and clinical knowledge [3], ChatGPT may pose risks to the integrity of online exams such as PREview. To our knowledge, and based on PubMed/Embase searches in May 2023, no studies have examined chatbot performance on SJTs or ethics exams. Hence, we evaluated the performance of GPT-3.5 and GPT-4 on the official AAMC PREview Practice Exam.
Activity
The AAMC granted permission for the usage of the AAMC PREview Practice Exam for this study. Two investigators independently categorized questions into AAMC’s eight pre-professional competencies using PREview’s scoring rationales; disagreements were mediated by discussion. All 186 items from the PREview Practice Exam were inputted into ChatGPT in May 2023 [2]. Using PREview exam instructions, we prompted ChatGPT to “rate the effectiveness” of several responses to given scenarios as either “very ineffective,” “ineffective,” “effective,” or “very effective.” Exam questions are freely available on the AAMC website (https://students-residents.aamc.org/media/13316/download).
We utilized validated SJT methodologies to calculate scores [3], since PREview’s precise algorithms were not accessible. Based on PREview’s scoring [2] and a similarly structured medical school SJT [4], we awarded full credit (+ 2 points) for correct answers (e.g., ChatGPT answered “effective,” and the rubric was marked “effective”), 1 point for similar answers (“effective”/ “very effective’), and 0 points for different answers (“effective”/ “very ineffective”). Non-answers were considered incorrect (0 points). We calculated raw scores based on total available points and adjusted final scores to PREview’s nine-point scale via normalization. As the mean raw score for the comparably graded SJT was 51.8% (33.17/64) [4], similar to the calculated mean PREview score of 53.8% (4.84/9) reported by the AAMC [5], our methodology likely approximates the true algorithm.
We then compared answers by model and competency using χ2 and Fisher’s exact tests. All analyses were conducted using Stata.
Results and Discussion
Raw and final scores were lower for GPT-3.5 (69.1%; 6/9) than for GPT-4 (76.9%; 7/9). Only 4% and 2% of questions were answered incorrectly by GPT-3.5 and GPT-4, respectively (Table 1). GPT-4 was more likely to be correct than GPT-3.5 (p = 0.017). GPT-3.5 did not describe any actions as “very effective,” answering “effective” more frequently (p < 0.001).
Table 1.
Comparison of PREview answers by chatbot model
| Answer | Points | GPT-3.5, n (%) | GPT-4, n (%) | Test statistic, p-value |
|---|---|---|---|---|
| Correct | + 2 | 78 (42%) | 103 (55%) | χ2 = 12.10, p = 0.017 |
| Partial | + 1 | 101 (54%) | 80 (43%) | |
| Incorrect | 0 | 7 (4%) | 3 (2%) | |
| Total | 186 | 186 | ||
| Very ineffective | – | 26 (14%) | 12 (6%) | χ2 = 118.87, p < 0.001 |
| Ineffective | – | 40 (22%) | 63 (34%) | |
| Effective | – | 120 (65%) | 57 (31%) | |
| Very effective | – | 0 (0%) | 54 (29%) |
Statistically significant relationships (p < 0.05) determined through χ2 tests are bolded
GPT-4 and GPT-3.5 performed similarly across competencies (Table 2). Notably, GPT-4 answered no questions incorrectly for five competencies, including ethics and social skills, and answered only one question incorrectly for three competencies.
Table 2.
Performance of GPT-3.5 and GPT-4 by pre-professional competency
| Competency (n) | GPT-3.5 | GPT-4 | p-Value | ||||
|---|---|---|---|---|---|---|---|
| Incorrect, n (%) | Partial, n (%) | Correct, n (%) | Incorrect, n (%) | Partial, n (%) | Correct, n (%) | ||
| Capacity for improvement | 0 (0.0) | 17 (16.8) | 16 (20.5) | 0 (0.0) | 11 (13.8) | 22 (21.4) | 0.28 |
| Cultural competence | 1 (14.3) | 6 (5.9) | 4 (5.1) | 1 (33.3) | 4 (5.0) | 6 (5.8) | 0.84 |
| Ethical responsibility to self and others | 0 (0.0) | 21 (20.8) | 22 (28.2) | 0 (0.0) | 17 (21.3) | 26 (2.5) | 0.76 |
| Reliability and dependability | 1 (14.3) | 13 (12.9) | 9 (11.5) | 1 (33.3) | 11 (13.8) | 11 (10.7) | 1.00 |
| Resilience and adaptability | 0 (0.0) | 10 (9.9) | 1 (1.3) | 0 (0.0) | 5 (6.3) | 6 (5.8) | 0.46 |
| Service orientation | 0 (0.0) | 4 (4.0) | 5 (6.4) | 0 (0.0) | 4 (5.0) | 5 (4.9) | 1.00 |
| Social skills | 3 (42.9) | 11(10.9) | 12 (15.4) | 0 (0.0) | 17 (21.3) | 9 (8.7) | 0.085 |
| Teamwork | 2 (28.6) | 19 (18.8) | 9 (11.5) | 1 (33.3) | 11 (13.8) | 18 (17.5) | 0.25 |
| Total | 7 (4) | 101 (54) | 78 (42) | 3 (2) | 80 (43) | 103 (55) | 0.009 |
Statistically significant relationships (p < 0.05), determined through Fisher’s exact tests, are bolded
ChatGPT outperformed the average medical school applicant on PREview as reported by the AAMC.5 Answering < 5% of questions incorrectly, GPT-3.5 and GPT-4 scored in the 78th and 93rd percentiles, respectively [5]. Although ChatGPT lacks an understanding of human emotions or ethics, we report that ChatGPT is capable of applying ethical principles in medicine, which is important for its potential deployment in healthcare. Interestingly, GPT-3.5 refused to strongly endorse actions as “very effective,” which may reflect inherent programming limitations. GPT-4’s stronger performance suggests that chatbot ethical reasoning capabilities have advanced.
Inaccessible PREview scoring algorithms required us to use established SJT scoring methodologies [4]. Although our estimates are likely accurate due to comparable scoring outcomes for prior SJTs and PREview [5], discrepancies may exist. Additionally, our analysis did not evaluate the reliability of ChatGPT’s answers by running exams multiple times. Our study is also limited by its cross-sectional design. Given rapid technological advances, future chatbot models may demonstrate stronger SJT performance. Ultimately, ChatGPT’s ability to correctly answer SJT questions is noteworthy regardless of scoring methodology.
ChatGPT’s capabilities now include ethical and professional reasoning comparable or superior to medical school applicants. These findings are critical for AI’s integration into healthcare, including possible usage of chatbot technologies to simulate scenarios that practicing physicians might encounter. However, they also present risks for online exam delivery. In particular, AAMC’s PREview exam is delivered virtually and may be subject to integrity issues. Future research is warranted to investigate the usage of artificial intelligence technology by medical students, trends in ethical and professional decision-making capabilities of chatbot models over time, and comparison with human performance on similar exams.
Author Contribution
Sai Kurapati and John Lin contributed equally as co-first authors.
Funding
John Lin was awarded departmental funding from Brown University for expenses related to this study.
Data Availability
The PREview Practice Exam is the property of the AAMC and is available at https://students-residents.aamc.org/media/7956/download.
Declarations
Conflict of Interest
The authors declare no competing interests.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
John C. Lin and Sai S. Kurapati contributed equally to this work.
References
- 1.Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on medical challenge problems. 2023. 10.48550/arXiv.2303.13375.
- 2.Association of American Medical Colleges. About the AAMC PREviewTM professional readiness exam. Assoc Am Med Coll. 2023. Accessed 7 May 2023. https://students-residents.aamc.org/aamc-preview/about-aamc-preview-professional-readiness-exam.
- 3.Webster ES, Paton LW, Crampton PES, Tiffin PA. Situational judgement test validity for selection: a systematic review and meta-analysis. Med Educ. 2020;54(10):888–902. doi: 10.1111/medu.14201. [DOI] [PubMed] [Google Scholar]
- 4.Husbands A, Rodgerson MJ, Dowell J, Patterson F. Evaluating the validity of an integrity-based situational judgement test for medical school admissions. BMC Med Educ. 2015;15(1):144. doi: 10.1186/s12909-015-0424-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.AAMC. Using AAMC PREview data in 2023 medical student selection. 2022. https://www.aamc.org/media/61226/download. Accessed 8 May 2023.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The PREview Practice Exam is the property of the AAMC and is available at https://students-residents.aamc.org/media/7956/download.