Unveiling the Potential of AI in Plastic Surgery Education: A Comparative Study of Leading AI Platforms’ Performance on In-training Examinations

Abstract

Background:

Within the last few years, artificial intelligence (AI) chatbots have sparked fascination for their potential as an educational tool. Although it has been documented that one such chatbot, ChatGPT, is capable of performing at a moderate level on plastic surgery examinations and has the capacity to become a beneficial educational tool, the potential of other chatbots remains unexplored.

Methods:

To investigate the efficacy of AI chatbots in plastic surgery education, performance on the 2019–2023 Plastic Surgery In-service Training Examination (PSITE) was compared among seven popular AI platforms: ChatGPT-3.5, ChatGPT-4.0, Google Bard, Google PaLM, Microsoft Bing AI, Claude, and My AI by Snapchat. Answers were evaluated for accuracy and incorrect responses were characterized by question category and error type.

Results:

ChatGPT-4.0 outperformed the other platforms, reaching accuracy rates up to 79%. On the 2023 PSITE, ChatGPT-4.0 ranked in the 95th percentile of first-year residents; however, relative performance worsened when compared with upper-level residents, with the platform ranking in the 12th percentile of sixth-year residents. The performance among other chatbots was comparable, with their average PSITE score (2019–2023) ranging from 48.6% to 57.0%.

Conclusions:

Results of our study indicate that ChatGPT-4.0 has potential as an educational tool in the field of plastic surgery; however, given their poor performance on the PSITE, the use of other chatbots should be cautioned against at this time. To our knowledge, this is the first article comparing the performance of multiple AI chatbots within the realm of plastic surgery education.

Takeaways

Question: How do popular artificial intelligence (AI) platforms perform on the Plastic Surgery In-service Training Examination (PSITE)?

Findings: ChatGPT-4.0 outperformed other AI platforms, ranking in the 95th percentile of first-year residents on the 2023 PSITE and reaching accuracy rates up to 79%.

Meaning: ChatGPT-4.0 has potential as an educational tool in the field of plastic surgery; however, given their poor performance on the PSITE, the use of other chatbots in plastic surgery education should be cautioned against at this time.

INTRODUCTION

Artificial intelligence (AI) chatbots have sparked fascination for their potential as an educational tool. Chatbots, also referred to as large language models (LLMs), are taught using extensive data sets and are trained to recognize patterns, giving them the ability to complete human-like tasks, such as engage in text conversation, answer questions, brainstorm ideas, and respond to writing prompts.^1–5 ChatGPT was one of the first publicly available LLMs, and its utility in medical education has been promising based on its passing performance on standardized examinations, including the US Medical Licensing Examination.^6,7 Gupta et al⁸ sought to test ChatGPT’s efficacy as an educational aid for plastic surgery residents, discovering that the platform could answer 2022 Plastic Surgery In-service Training Examination (PSITE) questions with an accuracy of 54.96%. Similarly, Humar et al⁹ reported that ChatGPT scored 57% on the 2022 PSITE, which would place the chatbot in the 49th percentile for first-year integrated plastic surgery residents. Although it is well documented that ChatGPT is capable of performing at a moderate level on the PSITE and has the capacity to be a beneficial educational tool, the potential of newer chatbots remains unexplored.

This study aims to evaluate and compare the performance of seven popular chatbots on the PSITE: ChatGPT-3.5, ChatGPT-4.0, Google Bard, Google PaLM, Microsoft Bing AI, Claude, and My AI by Snapchat. To our knowledge, this is the first article comparing the performance of multiple AI chatbots within the realm of plastic surgery education.

METHODS

Question Selection

To investigate AI’s role in plastic surgery education, we tested the performance of various AI platforms on the PSITE, adapting our protocol from Gupta et al.⁸ The PSITE is given to plastic surgery residents annually to assess knowledge across five disciplines: comprehensive plastic surgery, hand and lower extremity surgery, craniomaxillofacial surgery, aesthetic/cosmetic surgery, and core surgical principles.¹⁰ Examination scores allow academic programs to compare their students’ performance against their peers and can be used to gauge readiness for the American Board of Plastic Surgery written examination.^10,11 As such, we determined that chatbots’ PSITE scores would be an effective indicator of their plastic surgery knowledge.

Examinations and answer keys from the last 5 years (2019–2023) were obtained from the American Society of Plastic Surgeons (ASPS). As most AI chatbots are limited to processing text-based inputs, questions requiring an image or table to reach an appropriate answer were excluded unless the correct answer could be reasonably derived with solely text-based information. This determination was made depending on whether the answer key’s explanation referred to the image as part of its reasoning and whether the image was described in appropriate detail in the question stem.

AI Testing

We investigated the PSITE performance of seven AI chatbots: ChatGPT-3.5, ChatGPT-4.0, Google Bard, Google PaLM, Microsoft Bing AI, Claude, and My AI by Snapchat. Before entering questions into the platforms, we provided a prompt that asked the chatbots to “answer the following multiple-choice question and provide an explanation.” The prompt was input before each question for consistency.

Data Analysis

We recorded the number of questions answered correctly by each AI platform per exam using the ASPS PSITE answer key. These data were then analyzed using analysis of variance and covariance tests with Stata 15.1 (Stata Corp, 2017, Stata Statistical Software: Release 15; Stata Corp LLC, College Station, Tex.). Analyses were also run using ChatGPT-4.0 to gauge its computational ability. A P value of less than 0.05 was considered statistically significant.

Additionally, we conducted an in-depth analysis of responses produced by ChatGPT-4.0 on the 2023 PSITE. Each question was classified into one of the following categories: anatomy, pathophysiology, clinical recall, or clinical reasoning. For questions answered incorrectly, the responses were evaluated to identify the root cause of the inaccuracy and categorized as follows:

1. Factual inaccuracy: the answer justification contained false scientific information.

2. Use of outdated information: the answer was based off of outdated guidelines, publications, or scientific beliefs.

3. Improper consideration of the clinical vignette: the answer justification failed to incorporate a crucial piece of information from the question stem, leading to the wrong answer choice; however, the information provided was factually sound.

4. Logical fallacy: the correct answer was explained in the reasoning but the chatbot failed to select the correct multiple-choice option.

RESULTS

After excluding questions reliant on tables or images, the number of usable PSITE questions per exam varied by year: 238 (2019), 239 (2020), 228 (2021), 234 (2022), and 236 (2023). The percentage of correctly answered questions was calculated for each chatbot and examination year (Table 1). Across all examination years, the highest accuracy was 79% on the 2020 PSITE by ChatGPT-4.0, whereas the lowest score, 45%, was obtained by Google PaLM on the 2019 examination. Average AI performance on the PSITE was calculated using individual scores from the 2019–2023 examinations. Of the AI platforms investigated, ChatGPT-4.0 performed the strongest, obtaining an average score of 71.8 ± 4.5%. Average PSITE performance was comparable between the other platforms: ChatGPT-3.5 (52.6 ± 5.4%), Google Bard (57.0 ± 4.3%), Microsoft Bing AI (55.8 ± 4.4%), My AI by Snapchat (56.2 ± 6.1%), Google PaLM (48.6 ± 2.3%), and Claude (55.0 ± 3.7%) (Fig. 1). ANOVA multifactor analysis using overall raw scores for all five examination years revealed that performance differences among the AI platforms was statistically significant (P < 0.01).

Table 1.

Scores of AI Chatbots on the 2019–2023 PSITE (P < 0.01)

	ChatGPT 3.5, %	Google Bard, %	Microsoft Bing AI, %	My AI by Snapchat, %	ChatGPT 4.0, %	Google PaLM, %	Claude+, %
2019	46	57	51	47	68	45	50
2020	57	62	56	64	79	49	54
2021	52	55	55	58	73	50	60
2022	49	51	54	55	68	48	54
2023	59	60	63	57	71	51	57

Fig. 1.

Average AI chatbot performance on the 2019–2023 PSITE.

Two versions of the ChatGPT platform from OpenAI were utilized in this study. The newer model, ChatGPT-4.0, performed better than its older counterpart to a statistically significant degree (P < 0.01). The same is true when comparing the two chatbots created by Google; Google Bard outperformed Google PaLM (P = 0.01).

Average scores for the five subsections of the PSITE were calculated for each chatbot. All platforms reached their highest average on the core surgical principles section (ChatGPT-3.5 58.8%, Google Bard 61.2%, Microsoft Bing AI 66.2%, My AI by Snapchat 59%, ChatGPT-4.0 81.2%, Google PaLM 53.6%, and Claude 63.2%). Statistical significance between sections was found for all platforms except Google Bard and My AI by Snapchat (P = 0.20 and P = 0.08, respectively). For LLMs whose training database ended in 2021 (ChatGPT-3.5, ChatGPT-4.0, and Claude), no statistical significance was noted when comparing examination scores before and after the training cutoff (P = 0.70, P = 0.43, and P = 0.85, respectively).

Further investigation into ChatGPT-4.0’s performance on the 2023 PSITE demonstrated that the platform performed substantially worse on anatomy questions compared with other question types; it correctly answered 77% of clinical reasoning questions, 72% of clinical recall questions, 72% of pathophysiology questions, and 42% of anatomy questions. Analysis of answer justifications revealed that 79% of incorrect answers can be attributed to factual inaccuracy, 9% to logical fallacy, 6% to improper consideration of the clinical vignette, and 6% to the use of outdated information. Due to copyright restrictions, we are unable to provide specific answer examples.

Statistical analyses were also input into ChatGPT-4.0 to examine its computational ability. Upon entering the data and asking the chatbot to conduct an ANOVA analysis, ChatGPT-4.0 informed the user that it is unable to perform complex mathematical equations and instructed the user to use Python. However, with various prompt manipulation, we were able to elicit results from ChatGPT-4.0. Of the 13 ANOVA tests conducted, ChatGPT-4.0’s determination of statistical significance aligned with that of Stata 15.1 69% of the time, although the chatbot could not provide a numerical P value (Table 2).

Table 2.

Statistical Analyses Conducted on Stata 15.1 and ChatGPT-4.0

	Statistically Significant Result? (P < 0.05)
ANOVA Analysis	Stata 15.1	ChatGPT 4.0
PSITE performance between the different AI platforms	Yes	Yes
PSITE performance between ChatGPT 3.5 and ChatGPT 4.0	Yes	Yes
PSITE performance between Google PaLM and Google Bard	Yes	Yes
PSITE performance between subsections by ChatGPT 3.5	Yes	No
PSITE performance between subsections by Google Bard	No	Yes
PSITE performance between subsections by Microsoft Bing AI	Yes	Yes
PSITE performance between subsections by My AI by Snapchat	No	Yes
PSITE performance between subsections by ChatGPT 4.0	Yes	No
PSITE performance between subsections by Google PaLM	Yes	Yes
PSITE performance between subsections by Claude+	Yes	Yes
PSITE performance between pre-2021 and post-2021 examinations by ChatGPT 3.5	No	No
PSITE performance between pre-2021 and post-2021 examinations by ChatGPT 4.0	No	No
PSITE performance between pre-2021 and post-2021 examinations by Claude+	No	No

Of the 13 ANOVA tests conducted, the platforms yielded the same determination of statistical significance 69% of the time.

DISCUSSION

Proliferation of AI Chatbots

AI has captured the attention of the medical community for its potential to drastically transform the future of healthcare. Technology that once seemed like a distant possibility is now a reality, and if utilized properly, presents endless opportunities to benefit physicians and patients alike. Chatbots, or LLMs, are a form of AI designed to imitate human conversation; these platforms can process substantial amounts of data and use pattern recognition to answer questions or respond to prompts, rapidly accomplishing these tasks with little to no cost.^1,2 As the first chatbot of its kind, ChatGPT-3.5 has dominated much of the medical literature. However, since its release in 2022, various other AI chatbots have been developed and are rising in popularity, their potential remaining undiscovered.

Among the new AI chatbots are ChatGPT-4.0, Microsoft Bing AI, My AI by Snapchat, Claude, Google PaLM, and Google Bard. ChatGPT-4.0 is the newer model of ChatGPT-3.5 that is operated by OpenAI (San Francisco, Calif.). According to the company, the new model is slated to be “more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5.”³ Interestingly, OpenAI’s technology was utilized in the creation of Microsoft Bing AI and My AI by Snapchat, with additional modifications made by the companies.^12,13 Claude is an independent AI model created and operated by Anthropic (San Francisco, Calif.) and both Google chatbots, PaLM and Bard, are run by Google (Mountain View, Calif.).^4,5,14,15

Performance of Chatbots on the PSITE

Results of our study indicate that ChatGPT-4.0 has potential as an educational tool in the field of plastic surgery; however, the use of other chatbots in plastic surgery education should be cautioned against unless they are improved. Of the chatbots tested, ChatGPT-4.0 outperformed its competitors, reaching scores up to 79% on the PSITE. The performance among other chatbots was comparable, with average PSITE score (2019–2023) ranging from 48.6% to 57.0% (Fig. 1).

Although there is no passing score for the PSITE, chatbot performance can be compared with human performance using ASPS resident norms (Table 3). On the 2023 PSITE, ChatGPT-4.0 performed remarkably well compared with first-year residents, ranking in the 95th percentile. However, the chatbot’s percentile ranking subsequently declined when compared with residents further along in their education, scoring in the 82nd percentile of second year, 57th percentile of third year, 40th percentile of fourth year, 34th percentile of fifth year, and 12th percentile of sixth year.¹⁶ The performance of the other chatbots was unimpressive. Google PaLM fared the worst compared with residents, ranking in only the 10th percentile of first year, the third percentile of second year, and the 0th percentile for third through sixth years.¹⁶ Microsoft Bing and Google Bard performed moderately on the 2023 PSITE compared with first-year residents (63rd and 40th percentile, respectively), whereas ChatGPT-3.5, My AI by Snapchat, and Claude had poorer outcomes (32nd, 25th, 25th, respectively); none of these platforms surpassed the second percentile when compared with fifth- and sixth-year residents.¹⁶ Percentile ranking for all platforms worsened when compared with upper-level residents, suggesting that the use of AI chatbots as an educational resource is most beneficial to physicians early in their career. Of the chatbots analyzed, ChatGPT-4.0 would be the most suitable platform for this purpose, as it is the only chatbot whose performance ranked in the top quartile of residents.

Table 3.

AI Performance on the 2023 PSITE Compared with Residents in Integrated Programs

		Percentile Ranking Compared with Residents
	Exam Score (% Correct)	First Year	Second Year	Third Year	Fourth Year	Fifth Year	Sixth Year
ChatGPT-4	71	95th	82nd	57th	40th	34th	12th
Microsoft Bing AI	63	63rd	33rd	12th	11th	2nd	1st
Google Bard	60	40th	23rd	6th	4th	1st	1st
ChatGPT-3.5	59	32nd	17th	6th	2nd	1st	1st
My AI by Snapchat	57	25th	10th	3rd	1st	1st	0th
Claude+	57	25th	10th	3rd	1st	1st	0th
Google PaLM	51	10th	3rd	0th	0th	0th	0th

Adapted from American Society of Plastic Surgeons.¹⁶

Further analysis into ChatGPT-4.0’s performance on the 2023 PSITE revealed that the platform struggled most with anatomy questions (42% correct) compared with clinical recall (72%), pathophysiology (72%), and clinical reasoning (77%) questions. It is possible that this finding is related to the chatbot’s inability to work with image-based data, as anatomy is largely a visual-dependent field. Although plastic surgery residents should consult more traditional resources for anatomical questions, ChatGPT-4.0’s performance on the clinical and pathophysiologic questions is promising.

Analysis of ChatGPT-4.0’s justifications for incorrect answers on the 2023 PSITE revealed that the majority of errors were caused by factual inaccuracy (79%) rather than logical fallacy (9%), the use of outdated information (6%), or improper consideration of the clinical vignette (6%). This indicates that the program possesses the ability to think through complex problems and answer correctly, given it has access to the proper knowledge base. The low percentage of questions missed due to outdated information is particularly interesting, given that ChatGPT-4.0 only contains a knowledge base through 2021, whereas other platforms can harness up-to-date information through web searches.^3,17 The lack of post-2021 training does not seem to inhibit the platform’s potential in plastic surgery education, especially considering that changes in the field often take years to develop.

Finally, we wanted to assess whether ChatGPT-4.0 outperformed its predecessor, ChatGPT-3.5; OpenAI grants free access to ChatGPT-3.5, but charges $20 per month for ChatGPT-4.0. Analysis revealed that ChatGPT-4.0 performed drastically better than ChatGPT-3.5 on the PSITE, which is consistent with previous studies comparing their performance on neurosurgery and general surgery board examinations.^18,19

Utilization of AI Chatbots in Plastic Surgery Education

Although ChatGPT-4.0 has demonstrated its potential as an educational resource in plastic surgery based on its PSITE performance, there remains room for improvement, especially for the other chatbots investigated in this study. Considering many of these chatbots were only released within the last few years and newer models have already been developed, there is significant hope that the technology will progress at an accelerated pace in upcoming years. Although widespread implementation has not yet occurred, there are numerous ways in which AI chatbots can be incorporated into plastic surgery education:

1. Study resource for plastic surgery residents: Many have suggested that chatbots could assume the role of a “personal tutor,” answering questions, providing feedback, and creating novel study resources for physicians in training (practice questions, clinical vignettes, etc).^1,20,21 Furthermore, chatbots can provide these services at little to no cost; the platforms analyzed in this study are available free-of-charge, except ChatGPT-4.0 ($20/month).

2. Research tool: Chatbots will likely become a valuable research tool for plastic surgery residents because their ability to process extensive amounts of data allows for unique pattern recognition and generation of innovative research questions.^22–24 Additionally, they have potential to assist in the literature review and rough draft writing process which, some have suggested, could afford physicians more time to focus on the clinical significance of their research projects, rather than on the more arduous tasks.^25–27

3. Patient medical resource: Chatbots can serve as an additional outlet for patient questions. Multiple studies have illustrated ChatGPT’s ability to answer common patient questions, ranging from drug–drug interactions to inquiries regarding the risks, benefits, and expectations of plastic surgery procedures.^28–34 This has potential to improve patient education, shorten consult times, and increase patient satisfaction with their physician.

4. Clinical applications: Although still in the early stages, some researchers have attempted harnessing AI in the clinical realm, using the technology to assist with diagnostics, risk stratification, and surgical planning.^35–37 Additionally, there is hope that AI could alleviate the time-consuming administrative work often tasked to residents (operative notes, data collection, and discharge summaries), allowing them more time to focus on direct patient care.^38,39 Finally, AI has been tested as an objective assessment tool for surgical outcomes and technique, which may have potential to evaluate resident performance in a less biased manner.^40–42

Limitations of AI Chatbot Use in Plastic Surgery Education

Although chatbots have significant potential within plastic surgery, it is paramount that their limitations are recognized to ensure responsible usage:

1. Restricted usefulness for statistical analysis: Despite their ability to assist in research question formulation and study design, chatbots may be confined in their ability to conduct quantitative analysis. In this study, we ran statistical analyses through ChatGPT-4.0 after conducting the tests using Stata 15.1. ChatGPT-4.0 incorrectly determined statistical significance 31% of the time and was unable to provide specific P values. This limitation is recognized by the chatbot itself, which informs users that it is not adequately trained to perform complex mathematical equations and instead provides instructions on how to utilize other programming platforms, such as Python. Although proper manipulation of prompts input into ChatGPT-4.0 can ultimately yield statistical analyses, it is not able to replace an experienced statistician, and plastic surgery residents would likely obtain better results through mathematical programs like Python or Stata.

2. Fabricated references: Perhaps the most concerning drawback of AI is its well-documented tendency of fabricating references for the information it generates, providing references unrelated to the topic of discussion, and incorrectly citing references that do exist.^43–46 This is a significant limitation for those wanting to use AI in research or trace back information provided by chatbots to their original source.

3. Information bias: Chatbots’ ability to recognize patterns and make predictions is based on their data set training; if these data were biased, AI responses will be as well.^25–27,47 If AI chatbots are implemented in diagnostics or treatment planning, there is the possibility of patient endangerment if the chatbot is reliant on biased or out-of-date information.

4. Information inaccuracies: As witnessed by the moderate to poor performance of chatbots in this study, AI chatbots are not infallible. Although able to generate responses and explain their reasoning, the information provided may be inaccurate.^47,48 Even the strongest performing platform, ChatGPT-4.0, had an average PSITE score of 71.8 ± 4.5% and of incorrect responses on the 2023 PSITE, 79% were attributed to factual inaccuracy. Even though results of the present study seem to suggest newer chatbots are improving in their information accuracy, those wishing to utilize AI in plastic surgery should proceed cautiously, utilizing more traditional resources to fact-check and serve as ultimate authority. Revision to AI chatbots is needed to avoid the distribution of false information that can mislead trainees and patients alike.

5. Risk of plagiarism: Finally, AI chatbots present the risk of plagiarism. If utilized in the composition of research articles, practice questions, or literature reviews, the material must be subject to further human review to avoid infringing on others’ intellectual property.⁴⁸ Furthermore, ChatGPT is unable to fulfill authorship criteria, making its use in drafting materials for publication ethically ambiguous and a topic of debate that will likely escalate as its use becomes more widespread.⁴⁹

CONCLUSIONS

AI chatbots have the potential to revolutionize plastic surgery education. Although most chatbots lack proficiency in plastic surgery, the performance of ChatGPT-4.0 was encouraging for those wishing to harness AI as an educational resource in the field; the chatbot ranked in the 95th percentile of first-year residents on the 2023 PSITE and had accuracy rates up to 79%. Analysis revealed that the utility of AI as an educational resource seems greatest at the onset of residency training, and despite its higher price, ChatGPT-4.0 should be utilized over ChatGPT-3.5 due to its significantly enhanced proficiency in plastic surgery. Although ChatGPT-4.0 currently demonstrates promise as an educational tool, further refinements must be made before its use becomes widely implemented.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.