Skip to main content
NCBI home page
JBJS Open Access logoLink to JBJS Open Access
. 2025 Sep 11;10(3):e25.00112. doi: 10.2106/JBJS.OA.25.00112

ChatGPT-4o is Not a Reliable Study Source for Orthopaedic Surgery Residents

Neil Jain 1, Caleb Gottlich 2, John Fisher 2, Travis Winston 2, Kristofer Matullo 1, Dustin Greenhill 1,a
PMCID: PMC12417002  PMID: 40927594

Abstract

Background:

The use of artificial intelligence platforms by medical residents as an educational resource is increasing. Within orthopaedic surgery, older Chat Generative Pre-trained Transformer (ChatGPT) models performed worse than resident physicians on practice examinations and rarely answered questions with images correctly. The newer ChatGPT-4o was designed to improve these deficiencies but has not been evaluated. This study analyzed (1) ChatGPT-4o's ability to correctly answer Orthopaedic In-Training Examination (OITE) questions and (2) the educational quality of the answer explanations that it presents to our orthopaedic surgery trainees.

Methods:

The 2020 to 2022 OITEs were uploaded into ChatGPT-4o. Annual score reports were used to compare the chatbot's raw score with that of ACGME-accredited orthopaedic residents. ChatGPT-4o's answer explanations were then compared with those provided by the American Academy of Orthopaedic Surgeons (AAOS) and categorized based on (1) the chatbot's answer (correct/incorrect) and (2) the chatbot's answer explanation when compared with the explanation provided by AAOS subject-matter experts (classified as consistent, disparate, or nonexistent). Overall ChatGPT-4o response quality was then simplified into 3 groups. An “ideal” response combined a correct answer with a consistent explanation. “Inadequate” responses provided a correct answer but no explanation. “Unacceptable” responses provided an incorrect answer or disparate explanation.

Results:

ChatGPT-4o scored 68.8%, 63.4%, and 70.1% on the 2020, 2021, and 2022 OITEs, respectively. These raw scores corresponded with ACGME-accredited postgraduate year-5 (PGY-5), PGY2-3, and PGY-4 resident physicians. Pediatrics and Spine were the only subspecialties whereby ChatGPT-4o consistently performed better than a junior resident (≥PGY-3). The quality of responses provided by ChatGPT-4o was ideal, inadequate, or unacceptable in 58.7%, 6.9%, and 34.4% of questions, respectively. ChatGPT-4o scored significantly lower on media-related questions when compared with nonmedia questions (60.0% versus 73.1%, p < 0.001).

Conclusions:

ChatGPT-4o performed inconsistently on the OITE. Moreover, the responses it provided trainees were not always ideal. Its limited performance on media-based orthopaedic surgery questions also persisted. The use of ChatGPT by resident physicians while studying orthopaedic surgery concepts remains unvalidated.

Level of Evidence:

Level IV. See Instructions for Authors for a complete description of levels of evidence.

Introduction

The artificial intelligence (AI) chatbot called Chat Generative Pre-Transformer (ChatGPT) rapidly became one of the most popular software applications in history1-6. It gained confidence within the medical field after its older models demonstrated passing scores on the US Medical Licensing Examination and corresponding specialty examinations for multiple professions7-9. Within orthopaedic surgery, ChatGPT has shown utility by improving readability of patient education materials10-14, offering on-demand diagnostic assistance15-18, and answering clinical queries19-23.

More recently, ChatGPT has gained traction as a study aid for resident physicians given its ability to accept difficult questions (as inputs) and instantly provide answers with explanations (as outputs)24. Given that exposure to “as many practice questions as you can” is emphasized as a study tactic during board preparation, it is easy to understand why young trainees may use an easily accessible source like ChatGPT25. However, the inability for users to identify when this exceedingly popular (yet unvetted) software is teaching them something right, versus wrong, is unsettling. Massey et al. recently concluded that older ChatGPT versions (ChatGPT-3.5 and, its adjacent update, ChatGPT-4) performed worse than orthopaedic residents on American Academy of Orthopaedic Surgeons (AAOS) ResStudy questions and could only answer approximately one third of media-based questions (i.e. those with pictures or videos) correctly1. However, they acknowledged that ChatGPT-4 at least performed better than the older ChatGPT-3.5. Kung et al. similarly concluded that ChatGPT-4 performed better on combined OITE questions [superior to a postgraduate year-5 (PGY-5) resident] than ChatGPT-3.5 (which ranked worse than a PGY-2 resident)26. However, they raised concerns that only 47.2% of question explanations cited a verifiable source. The newer ChatGPT-4o (“o” for “omni”) model, released in mid-2024, advertised substantially improved media comprehension and modernized knowledge updates until at least October 202327,28. Given these enhancements, ChatGPT-4o might have improved abilities with which to answer advanced orthopaedic inquiries. However, the capabilities of this upgraded model within the orthopaedic surgery training environment have not been formally assessed.

This study aims to analyze ChatGPT-4o's ability to (1) answer Orthopaedic In-Training Examination (OITE) questions correctly and (2) appropriately explain those answers to orthopaedic surgery trainees.

Materials and Methods

All 635 published OITE questions from 2020 to 2022 were sorted by 2 blinded authors into the 11 distinct orthopaedic knowledge domains specified by the AAOS29. Disputes in classification were resolved by an orthopaedic surgeon certified by the American Board of Orthopaedic Surgery.

A ChatGPT Plus subscription was obtained. Individual OITE questions were then input into ChatGPT-4o with all associated media (images, figures, or short videos). Each input was entered using a new chatbox between queries to avoid ChatGPT-4o memory retention or user feedback7. Overall ChatGPT-4o response quality was determined by considering whether the answer choice was simply correct/incorrect in addition to the chatbot’s answer explanation (Fig. 1).

Fig. 1.

Fig. 1

Open in a new tab

Categorization Method Used to Summarize ChatGPT-4o's Overall Response Quality

Binary (correct/incorrect) answer choices for each question were recorded. Overall ChatGPT-4o raw scores per examination year were compared with average individual percentile ranks of ACGME-accredited orthopaedic surgery residents using annual OITE score reports published by the AAOS. Of note, OITE blueprints after 2020 did not include “practice management” questions, thus preventing yearly comparisons in that domain30. In this study, residents typically in their first 2 years of orthopaedic surgery training (PGY-1, PGY-2) were considered junior residents and those typically in their third to fifth years of residency (PGY-3 to PGY-5) were considered senior residents.

To objectively categorize ChatGPT-4o's answer explanations, each AI-generated explanation was compared with the corresponding explanation provided by AAOS subject-matter experts and the OITE Examination Committee. An explanation was defined as “consistent” (C) if the chatbot’s identified pathology and reasoning paralleled the explanation provided by the AAOS. Matching source citations were not required. An explanation was considered “disparate” (D) if the identified diagnosis and/or explanation was completely different from the explanation provided by AAOS test writers. For example, a question that was categorized as having a correct answer but disparate explanation asked for the biological cause of radiographs demonstrating osteopetrosis. While ChatGPT-4o correctly answered “dysregulated osteoclast function”, its answer explanation disparately attributed the radiographic findings to and discussed Paget’s disease. For “nonexistent” (N) explanations, the chatbot answered the question but provided no explanation to justify its answer. All explanations were categorized (as C, D, or N) by a resident physician and a board-certified orthopaedic surgeon. When those raters disagreed, a third individual reviewed the explanation to resolve the conflict. There were no scenarios in which 2 of 3 raters did not agree. Cohen’s kappa statistic was calculated to test inter-rater reliability associated with explanation categories31.

ChatGPT-4o's overall response (correct/incorrect answer plus the chatbot’s explanation) were then organized into 6 categories using a modified version of previously established chatbot output logic groupings (Fig. 1)32. The 6 categories were two-letter acrophonical combinations based on (1) a correct (C) versus incorrect (I) answer choice and (2) an answer explanation that, when compared with AAOS test writers, was consistent (C), disparate (D), or nonexistent (N).

To more simply classify ChatGPT-4o's overall response quality per question, those 6 acrophonical categories were subgrouped as: ideal, inadequate, and unacceptable. An “ideal” response was defined as having a correct answer with a consistent explanation. In other words, “ideal” responses give trainees correct information with explanations similar to those written by AAOS subject-matter experts. “Inadequate” responses had correct answers but no explanations. “Unacceptable” responses were those whereby the chatbot either provided an incorrect answer or a disparate explanation (IC, IN, ID, CD). Trainees studying an “unacceptable” ChatGPT-4o response would learn incorrect information or reasoning.

Chi-squared or Fisher's exact tests were used for comparisons across groups. All data analyses were conducted using Stata (Version 18, StataCorp LLC). p-values <0.05 were considered statistically significant.

Results

ChatGPT-4o answered 67.4% correct among all 635 questions. Its yearly raw scores averaged 68.8%, 63.4%, and 70.1% for 2020, 2021, and 2022, respectively. These raw values corresponded to the individual percentile rank of ACGME-accredited PGY-5, PGY-2-3, and PGY-4 residents, respectively.

Table I subcategorizes answer correctness by examination year (for all questions and media-based questions) and knowledge domain. Answer correctness did not statistically differ by examination year among overall (p = 0.297) or media-based (p = 0.826) questions or knowledge domain among overall (p = 0.067) or media-based (p = 0.608) questions.

TABLE I.

Number of Correct Answers Per Examination Year and Knowledge Domain

Correct Answer
Yes No Total p
No. of questions 428 207 635
Questions per year 0.297
 2020 148 67 215
 2021 135 78 213
 2022 145 62 207
Media-based questions per year 0.826
 2020 50 36 86
 2021 63 43 106
 2022 52 31 83
OITE knowledge domain, n (%) 0.067
 Basic Science 32 (7.5) 8 (3.9) 40 (6.3)
 Foot and Ankle 33 (7.7) 29 (14.0) 62 (9.8)
 Hand and Wrist 34 (7.9) 20 (9.7) 54 (8.5)
 Hip and Knee 50 (11.7) 29 (14.0) 79 (12.4)
 Oncology 39 (9.1) 18 (8.7) 57 (9.0)
 Pediatrics 34 (7.9) 13 (6.3) 47 (7.4)
 Practice management 27 (6.3) 10 (4.8) 37 (5.8)
 Shoulder and Elbow 41 (9.6) 12 (5.8) 53 (8.4)
 Spine 53 (12.4) 24 (11.6) 77 (12.1)
 Sports 32 (7.5) 9 (4.3) 41 (6.5)
 Trauma 53 (12.4) 35 (16.9) 88 (13.9)
Open in a new tab

P-values reference: questions per year, media-based questions per year, and OITE knowledge domains.

The level of agreement (inter-rater reliability) among raters categorizing answer explanations was strong (к = 0.845). Answer explanations were consistent, disparate, or nonexistent in 62%, 28%, and 10% of OITE questions, respectively. Answer explanations among separate knowledge domains were not significantly different in the overall data set (p = 0.094) or among media-based questions (p = 0.148).

ChatGPT-4o's overall response quality varied, with 58.7% of all responses being ideal, 6.9% inadequate, and 34.3% unacceptable (Table II). Correct answers most often had consistent explanations (87.2%), and incorrect answers most often had poor (disparate or nonexistent) explanations (91.3%). The relationship between these variables (a correct or incorrect answer having either a consistent or poor explanation) was significant (p < 0.001).

TABLE II.

Categorizations and Associated Quality of ChatGPT-4o Responses

Overall Media-based Questions (Images, Figures, or Short Videos)
Quality of Response Response Category (%) Quality of Response Response Category (%)
Ideal CC 373 (58.7) Ideal CC 149 (54.2)
Inadequate CN 44 (6.9) Inadequate CN 5 (1.8)
Unacceptable (34.3%) CD 11 (1.7) Unacceptable (44.1%) CD 11 (4.0)
IC 18 (2.8) IC 12 (4.4)
IN 20 (3.2) IN 1 (0.4)
ID 169 (26.6) ID 97 (35.3)
Open in a new tab

CC = correct answer, consistent explanation, CN = correct answer, nonexistent explanation, CD = correct answer, disparate explanation, IC = incorrect answer, consistent explanation, IN = incorrect answer, nonexistent explanation, and ID = incorrect answer, disparate explanation.

ChatGPT-4o only demonstrated performance equivalent to or better than a PGY-3 resident every year in Pediatrics and Spine. However, within every other domain, ChatGPT-4o's performance was comparable with a junior resident or worse in at least one of the 3 years (Table III). Post hoc analysis did not demonstrate that Pediatrics and/or Spine had unanticipated proportions of text-only questions.

TABLE III.

ChatGPT-4o's Performance Categorized by Knowledge Domain

2020 2021 2022
OITE Knowledge Domain Score (%) PGY Level Score (%) PGY Level Score (%) PGY Level
Basic Science 90.1 >5 55.6 <1 77.8 >5
Foot and Ankle 47.8 1 50.0 <1 69.2 4
Hand and Wrist 69.2 >5 64.0 3 56.3 1-2
Hip and Knee 65.4 5 63.3 3 60.9 1-2
Oncology 66.7 4 83.3 >5 53.3 1-2
Pediatrics 76.9 >5 62.5 3-4 77.8 >5
Practice Management 33.3 3 86.7 n/a 68.4 n/a
Shoulder and Elbow 83.3 >5 56.5 1-2 100.0 >5
Spine 65.6 3 75.0 >5 68.0 4
Sports 78.6 >5 50.0 <1 94.1 >5
Trauma 66.7 4 52.4 1-2 58.8 <1
Open in a new tab

PGY = postgraduate year, and n/a = data unavailable.

Consistent demonstration ≥ PGY3 across all years in is given in bold.

When compared with nonmedia, the chatbot scored significantly lower on media-based questions (73.1% versus 60.0%, p < 0.001). In the media-based subset, yearly performance was 58.1%, 59.4%, and 62.7% for 2020, 2021, and 2022, respectively. On media-based questions, correct answers again most often had consistent explanations (90.3% versus 9.7%) and incorrect answers most often had disparate or nonexistent explanations (88.2% versus 11.8%). ChatGPT-4o’s output quality on media-based questions varied, with 54.2% of responses being ideal, 1.8% inadequate, and 44.1% unacceptable (Table II). Among 3 questions with videos (2 related to computed tomography / magnetic resonance imaging, 1 related to arthroscopy), ChatGPT-4o answered one correct.

Discussion

In this study, ChatGPT-4o's ability to answer and explain OITE questions to orthopaedic surgery trainees was deemed unacceptable in over a third of cases. While the improved raw performance of ChatGPT-4o versus previous models on media-based questions (60.0% correct versus 22.4%-35.7%, respectively) deserves some recognition, it still provided unacceptable responses to these question types 44.1% of the time.1

Our desire to classify ChatGPT-4o's overall answer quality (as ideal, inadequate, or unacceptable) originated from (1) recognition that most studies evaluating AI’s accuracy in orthopaedic education only assess whether it can answer examination questions right or wrong and (2) the reality that younger orthopaedic surgery trainees often use AI while studying practice questions. However, concerns regarding its consistency and trustworthiness with clinical information are emerging33-36. These concerns are pertinent to orthopaedic surgery resident education because study sources used by trainees are transitioning away from textbooks.37

The ability of AI to produce outputs desired by physicians is rapidly improving. In its first iteration, ChatGPT-3.5 was limited to its training on 175 billion parameters and contained knowledge of events only occurring before 202232,38. That model's OITE performance was similar to a PGY-1 orthopaedic surgery resident; however, fallacies in logic, stepwise thinking, or media analysis remained a limiting factor1,24,26,32. Its successor, ChatGPT-4, contained a larger base of information and enhanced abilities to solve complex problems39. As a result, ChatGPT-4 demonstrated major OITE performance improvements by scoring better than a PGY-5 resident26. However, those investigations of OITE performance did not include questions with media. After comparing resident-answered ResStudy question banks with ChatGPT-4 responses, Massey et al. found that ChatGPT-4 answered text-only questions correct almost twice as often as image-based questions (61.0% vs 35.7%, p < 0.001)1. Although ChatGPT-4 answered media-based questions better than ChatGPT-3.5 (35.7% vs 22.4%, respectively), there was clearly still room for improvement. Our study’s finding that the newer ChatGPT-4o correctly answers media-based questions almost twice as often as ChatGPT-4 (60.0% vs 35.7%, respectively) is another improvement worth acknowledgement.

Although AI may perform well in other fields40-42, ChatGPT-4o does not consistently answer well enough on orthopaedic examinations to be used as a primary study resource for orthopaedic surgery residents. It is not clear why, but may be because OITE questions are formatted as “4 or 5-part multiple-choice, with a single correct/best response key and 3 to 4 competing and compelling distractors (noncorrect options)”30. Despite its upgrades, we found that ChatGPT-4o continued to perform inconsistently, with its raw score ranging from an ACGME-accredited PGY-2 to PGY-5 level. While modest improvements in raw scores and other variables are encouraging, the inconsistency and our objective assessments of response quality are not. As previously stated, orthopaedic residents are supplementing traditional resources (i.e. textbooks) with easily accessible (i.e. mobile) resources such as question banks, online videos, or smartphone apps37,43,44. Our inability to predict when this exceedingly popular (yet unvetted) software will teach an orthopaedic surgery resident something correct, partially correct, or incorrect should be concerning.

Considering that ChatGPT-4o is marketed to have enhanced visual comprehension, its current performance of 58.1 to 62.7% on musculoskeletal media-based questions (although better than its predecessor ChatGPT-4) remains undesirable27,28,45. In their meta-analysis, Nowroozi et al. demonstrated that many other AI algorithms are capable of detecting fractures on plain radiographs with high rates of sensitivity and specificity46. It is unclear if the issues with ChatGPT’s performance on media-based orthopaedic surgery examination questions are due to the software, format of the examination, image quality, or other factors.

This study had limitations warranting discussion. First, OITE questions were classified into a single domain despite potentially covering multiple subject areas. This is a common limitation of assessing resident performance within specified domains as well. To ensure consistency in categorization, multiple authors had to agree on domain categorization. Second, the categorization of ChatGPT-4o's answer explanations (Fig. 1) was partially subjective. However, there was strong inter-rater reliability during the categorization process and we estimated that overall quantitative proportions would change <5% when another rater was included. Moreover, all questions grouped as CD were reviewed twice by a board-certified orthopaedic surgeon. Therefore, the addition of more raters is unlikely to alter our summative conclusions. Third, answer explanations categorized as “consistent” did not require matching source citations. Our methodology potentially masks a known problem in which a response we considered “ideal” does not share with our trainees a landmark, or even real, citation because AI is known to create fabricated references26,47. Had we required matching citations in our explanation criteria, the proportion of ideal overall responses would substantially decrease. It was our anecdotal experience during data collection that answer explanations provided by ChatGPT-4o were less educational than those written by the AAOS. Fourth, ChatGPT-4o's assessment of media-based questions required uploading files directly into its input box. Conversion of media files into suitable formats might have diminished their quality and affected the chatbot’s ability to interpret imaging. Fifth, the OITE represents only a single objective tool by which to assess ChatGPT-4o's ability to answer and explain orthopaedic examination questions. However, since 2020, the AAOS and American Board of Orthopaedic Surgery (ABOS) have collaborated to better align the OITE with ultimate board certification. Furthermore, the “extensive validation of each” OITE question “provides the foundation for content validity of the examination and supports the fundamental purpose of the OITE” as an educational tool30. Sixth, the OITE is a 275-question examination. Although 737 questions were not flagged between 2020 and 2022, our three-year data set only included 635 questions. Because of the collaborative effort between the AAOS and ABOS, some questions appearing on the OITE may be included on the ABOS Part I Certifying Examination and thus were not released after the examination30. The inputs we provided to ChatGPT-4o were all questions made available to residents. Finally, ChatGPT is a rapidly evolving technology. Rapid amendments to the chatbot are likely to produce different results that can be the subject of future investigations.

Conclusions

Across 3 years of OITE practice questions, ChatGPT-4o provided unacceptable responses with which to train future orthopaedic surgeons over one third of the time. Its raw score performance inconsistently ranged from a PGY-2 to PGY-5 level. Its ability to interpret media-based questions is better than that of previous models, but the accuracy and quality of its responses to media-based questions are still worse than questions without media. Sole dependence on ChatGPT-4o by resident physicians while studying orthopaedic surgery examination questions will often be unacceptable.

Footnotes

Investigation performed at St. Luke's University Health Network, Bethlehem, PA, and Texas Tech University Health Sciences Center, Lubbock, TX

Disclosure: The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article (http://links.lww.com/JBJSOA/A914).

Contributor Information

Neil Jain, Email: njain5466@gmail.com.

Caleb Gottlich, Email: Caleb.Gottlich@ttuhsc.edu.

John Fisher, Email: John.Fisher@ttuhsc.edu.

Travis Winston, Email: Travis.Winston@ttuhsc.edu.

Kristofer Matullo, Email: Kristofer.Matullo@sluhn.org.

References

  • 1.Massey PA, Montgomery C, Zhang AS. Comparison of ChatGPT–3.5, ChatGPT-4, and orthopaedic resident performance on orthopaedic assessment examinations. J Am Acad Orthop Surg. 2023;31(23):1173-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, Rizzo C. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023;11:1166120. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Eysenbach G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers. JMIR Med Educ. 2023;9:e46885. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Mesko B. The ChatGPT (generative artificial intelligence) revolution has made artificial intelligence approachable for medical professionals. J Med Internet Res. 2023;25:e48392. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Lubitz M, Latario L. Performance of two artificial intelligence generative language models on the orthopaedic In-Training examination. Orthopedics. 2024;47(3):e146-e150. [DOI] [PubMed] [Google Scholar]
  • 6.Özbek EA, Ertan MB, Kından P, Karaca MO, Gürsoy S, Chahla J. ChatGPT can offer at least satisfactory responses to common patient questions regarding hip arthroscopy. Arthroscopy. 2025;41(6):1806-27. [DOI] [PubMed] [Google Scholar]
  • 7.Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:e45312. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Liu M, Okuhara T, Chang X, Shirabe R, Nishiie Y, Okada H, Kiuchi T. Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis. J Med Internet Res. 2024;26:e60807. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.LaNicca M, Wright E, Lutnick E. Readability of orthopaedic patient educational material: an artificial intelligence application. J Clin Orthop Trauma. 2025;64:102971. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Busigó Torres R, Restrepo M, Stern BZ, Yahuaca BI, Buerba RA, García IA, Hernandez VH, Navarro RA. Artificial intelligence shows limited success in improving readability levels of spanish-language orthopaedic patient education materials. Clin Orthop Relat Res. 2025;483(7):1185-92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Chandra K, Ghilzai U, Lawand J, Ghali A, Fiedler B, Ahmed AS. Improving readability of shoulder and elbow surgery online patient education material with Chat GPT (chat generative pretrained transformer) 4. J Shoulder Elbow Surg. 2025. [DOI] [PubMed] [Google Scholar]
  • 13.Reaver CN, Pereira DE, Carrillo EV, Marcos CR, Goldfarb CA. Evaluating the performance of artificial intelligence for improving readability of online English- and spanish-language orthopaedic patient educational material: challenges in bridging the digital divide. J Bone Jt Surg. 2025;107(8):e36. [DOI] [PubMed] [Google Scholar]
  • 14.Abdelmalek G, Uppal H, Garcia D, Farshchian J, Emami A, McGinniss A. Leveraging ChatGPT to produce patient education materials for common hand conditions. J Hand Surg Glob Online. 2025;7(1):37-40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Dagher T, Dwyer EP, Baker HP, Kalidoss S, Strelzow JA; Dr. AI Will See You Now. How do ChatGPT-4 treatment recommendations align with orthopaedic clinical practice guidelines?. Clin Orthop Relat Res. 2024;482(12):2098-106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Hartman H, Essis MD, Tung WS, Oh I, Peden S, Gianakos AL. Can ChatGPT-4 diagnose and treat like an orthopaedic surgeon? Testing clinical decision making and diagnostic ability in soft-tissue pathologies of the foot and ankle. J Am Acad Orthop Surg. 2024;15. [DOI] [PubMed] [Google Scholar]
  • 17.Essis MD, Hartman H, Tung WS, Oh I, Peden S, Gianakos AL. Comparison of ChatGPT's diagnostic and management accuracy of foot and ankle bone-related pathologies to orthopaedic surgeons. J Am Acad Orthop Surg. 2025;10. [DOI] [PubMed] [Google Scholar]
  • 18.Liu J, Segal K, Daher M, Ozolin J, Binder WD, Bergen M, McDonald CL, Owens BD, Antoci V. Artificial intelligence versus orthopedic surgeons as an orthopedic consultant in the emergency department. Injury. 2025;56(4):112297. [DOI] [PubMed] [Google Scholar]
  • 19.Sparks CA, Fasulo SM, Windsor JT, Bankauskas V, Contrada EV, Kraeutler MJ, Scillia AJ. ChatGPT is moderately accurate in providing a general overview of orthopaedic conditions. JB JS Open Access. 2024;9(2):e23.00129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Shen OY, Pratap JS, Li X, Chen NC, Bhashyam AR. How does ChatGPT use source information compared with google? A text network analysis of online health information. Clin Orthop Relat Res. 2024;482(4):578-88. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Ghilzai U, Fiedler B, Ghali A, Singh A, Cass B, Young A, Ahmed AS. ChatGPT provides acceptable responses to patient questions regarding common shoulder pathology. Shoulder Elbow. 2024:17585732241283971. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Jagiella-Lodise O, Suh N, Zelenski NA. Can patients rely on ChatGPT to answer hand pathology-related medical questions? Hand (N Y). 2024:15589447241247246. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Li LT, Adelstein JM, Sinkler MA, Mistovich RJ. Artificial intelligence promotes the dunning kruger effect: evaluating ChatGPT answers to frequently asked questions about adolescent idiopathic scoliosis. J Am Acad Orthop Surg. 2025;33(9):473-80. [DOI] [PubMed] [Google Scholar]
  • 24.Mendiratta D, Herzog I, Singh R, Para A, Joshi T, Vosbikian M, Kaushal N. Utility of ChatGPT as a preparation tool for the orthopaedic In-Training examination. J Exp Orthopaedics. 2025;12(1):e70135. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Hamilton D. How to Prepare for the American Board of Orthopaedic Surgery Exam. AAOS Now; 2017. https://www.aaos.org/globalassets/education/courses/3041-new-board-prep/aaosnow_boardprep_web.pdf. Accessed July 2, 2025. [Google Scholar]
  • 26.Kung JE, Marshall C, Gauthier C, Gonzalez TA, Jackson JB, III. Evaluating ChatGPT performance on the orthopaedic In-Training examination. JB JS Open Access. 2023;8(3):e23.00056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.OpenAI. Hello GPT-4o. OpenAI. 2024. Available at: https://openai.com/index/hello-gpt-4o/. Accessed July 2, 2025. [Google Scholar]
  • 28.OpenAI. GPT-4 Overview. OpenAI platform. Available at: https://platform.openai.com/docs/models/gpt-4o. Accessed July 2, 2025.
  • 29.Le HV, Wick JB, Haus BM, Dyer GSM. Orthopaedic In-Training examination: history, perspective, and tips for residents. J Am Acad Orthop Surg. 2021;29(9):e427-e437. [DOI] [PubMed] [Google Scholar]
  • 30.AAOS . Orthopaedic In-Training Examination (OITE) Technical Report 2021. American Academy of Orthopaedic Surgeons; https://www.aaos.org/globalassets/education/product-pages/oite/oite-2021-technical-report.pdf. 2021, Accessed July 2, 2025. [Google Scholar]
  • 31.McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-82. [PMC free article] [PubMed] [Google Scholar]
  • 32.Jain N, Gottlich C, Fisher J, Campano D, Winston T. Assessing ChatGPT's orthopedic in-service training exam performance and applicability in the field. J Orthop Surg Res. 2024;19(1):27. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Li AW, Adelstein JM, Li LT, Sinkler MA, Mistovich RJ. Assessing ChatGPT responses to frequently asked questions regarding pediatric supracondylar humerus fractures. J Pediatr Orthop. 2025;45(6):327-31. [DOI] [PubMed] [Google Scholar]
  • 34.Megafu M, Guerrero O, Yendluri A, Parsons BO, Galatz LM, Li X, Kelly JDt, Parisien RL. ChatGPT and gemini are not consistently concordant with the 2020 American Academy of Orthopaedic Surgeons Clinical Practice Guidelines when evaluating Rotator Cuff Injury. Arthroscopy. 2025;41(8):2753-7. [DOI] [PubMed] [Google Scholar]
  • 35.Nwachukwu BU, Varady NH, Allen AA, Dines JS, Altchek DW, Williams RJ, 3rd, Kunze KN. Currently available large language models do not provide musculoskeletal treatment recommendations that are concordant with evidence-based clinical practice guidelines. Arthroscopy. 2025;41(2):263-75.e6. [DOI] [PubMed] [Google Scholar]
  • 36.Kim HJ, Yoon PW, Yoon JY, Kim H, Choi YJ, Park S, Moon JK. Discrepancies in ChatGPT's hip fracture recommendations in older adults for 2021 AAOS evidence-based guidelines. J Clin Med. 2024;13(19):5971. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Raja BS, Choudhury AK, Paul S, Rajkumar S, Kalia RB. Online educational resources for orthopaedic residency-a narrative review. Int Orthop. 2021;45(8):1911-22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.OpenAI. Introducing ChatGPT. OpenAI. 2022. Available at: https://openai.com/index/chatgpt/. Accessed July 2, 2025. [Google Scholar]
  • 39.OpenAI. GPT-4. Open AI. Available at: https://openai.com/product/gpt-4. Accessed July 2, 2025. [Google Scholar]
  • 40.Hubany S, Scala F, Hashemi K, Kapoor S, Fedorova J, Vaccaro M, Ridout R, Hedman C, Kellogg B, Leto Barone AA. ChatGPT-4 surpasses residents: a study of artificial intelligence competency in plastic surgery In-service examinations and its advancements from ChatGPT-3.5. Plast Reconstr Surg Glob Open. 2024;12(9):e6136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Skalidis I, Cagnina A, Luangphiphat W, Mahendiran T, Muller O, Abbe E, Fournier S. ChatGPT takes on the European exam in core cardiology: an artificial intelligence success story? Eur Heart J Digital Health. 2023;4(3):279-81. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Ali R, Tang OY, Connolly ID, Zadnik Sullivan PL, Shin JH, Fridley JS, Asaad WF, Cielo D, Oyelese AA, Doberstein CE, Gokaslan ZL, Telfeian AE. Performance of ChatGPT and GPT-4 on neurosurgery written board examinations. Neurosurgery. 2023;93(6):1353-65. [DOI] [PubMed] [Google Scholar]
  • 43.Margalit A, Mixa P, Day L, Marrache M, Mitchell S, Suresh KV, Wang K, Sabharwal S, Li TP, Loeb A, Naziri Q, Henn RF, III, Laporte D. Top three learning platforms for orthopaedic In-Training knowledge produce different results: JAAOS: Glob Res Rev. 2021;5(8):e21.00148. [Google Scholar]
  • 44.Rogers MJ, Zeidan M, Flinders ZS, Presson AP, Burks R. Educational resource utilization by current orthopaedic surgical residents: a nation-wide survey. J Am Acad Orthop Surg Glob Res Rev. 2019;3(4):e041. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Hayes DS, Foster BK, Makar G, Manzar S, Ozdag Y, Shultz M, Klena JC, Grandizio LC. Artificial intelligence in orthopaedics: performance of ChatGPT on text and image questions on a complete AAOS orthopaedic In-Training examination (OITE). J Surg Educ. 2024;81(11):1645-9. [DOI] [PubMed] [Google Scholar]
  • 46.Nowroozi A, Salehi MA, Shobeiri P, Agahi S, Momtazmanesh S, Kaviani P, Kalra MK. Artificial intelligence diagnostic accuracy in fracture detection from plain radiographs and comparing it with clinicians: a systematic review and meta-analysis. Clin Radiol. 2024;79(8):579-88. [DOI] [PubMed] [Google Scholar]
  • 47.Khan F. Regulating the revolution: a legal roadmap to optimizing AI in healthcare. Minn J L Sci Technol. 2023;25(1):49. [Google Scholar]

Articles from JBJS Open Access are provided here courtesy of Wolters Kluwer Health

RESOURCES