Abstract
Background: Although analyses of the Orthopaedic In-Training Examination (OITE) subspecialty content domains have been performed, few studies have analyzed the levels of evidence (LoEs) for journal articles used as references to create OITE questions. We present an analysis of reference characteristics and question taxonomy for the hand surgery content domain on the OITE. We aim to determine whether level of evidence (LoE) for hand surgery questions have increased over a 15-year period. Methods: All questions and references in the hand surgery content domain on the OITE from 1995-1997 and 2010-2012 were reviewed. The taxonomic classification was determined for each question. Publication characteristics were defined for each reference, and each primary journal article was assigned a LoE. Results: A total of 129 questions containing 222 references met inclusion criteria: 76 questions from 1995-1997 and 53 from 2010-2012. The Journal of Hand Surgery American and European Volumes, Journal of Bone and Joint Surgery American and British Volumes, and the Journal of the American Academy of Orthopaedic Surgeons were the most frequently cited journals overall. Recent examinations were more likely to have Buckwalter T3 complex clinical management questions. There was a statically significant increase in the LoE used to create hand questions on the 2010-2012 compared with the 1995-1997 OITE. Conclusions: Primary journal articles cited on the hand surgery content domain of the OITE frequently included recent publications from both general and subspecialty journals. More recent examination questions appear to test clinical management scenarios. LoE for hand questions has increased over a 15-year period. Our results can be used as a guide to help prepare orthopedic residents for the OITE.
Keywords: OITE, levels of evidence, resident education, evidence-based medicine
Background
The Orthopaedic In-Training Examination (OITE) is administered by the American Academy of Orthopaedic Surgeons (AAOS) to orthopedic surgery residents.11 OITE performance has been correlated with American Board of Orthopaedic Surgery (ABOS) Part 1 Examination performance.8 Hand surgery is 1 of the 11 content domains on the OITE and comprises around 8% of the examination.12
During the past 10 years, most peer-reviewed orthopedic journals have published the level of evidence (LoE) for original clinical articles.7,18,22 Many of these journals have made an effort to improve the LoE for published articles.5,16 Few studies in the literature have specifically addressed the hand surgery content domain on the OITE. In their analysis of clinical management questions on the OITE, Bernstein et al noted that more than two-thirds of these questions were supported by lower LoE.1 More recent authors have also concluded that the majority of references cited on the 2010-2012 OITE were level IV or level V evidence.6
We provide an analysis of reference characteristics used to create questions for the hand surgery content domain of the OITE from 1995-1997 and 2010-2012 as well as an analysis of question taxonomy. We aim to determine whether LoE for primary journal articles used as references for hand surgery questions on the OITE have increased over a 15-year period.
Materials and Methods
All questions in the hand surgery content domain on the OITE from 1995-1997 and from 2010-2012 were included. We omitted all questions that were excluded from official OITE scoring.
We reviewed the list of references provided with the OITE score report for each question and recorded the year that each reference was published. Each reference was categorized as a primary journal article, a journal review article, a textbook, or an instructional course lecture. Three authors reviewed each article and the LoE for each was determined in accordance with the AAOS Levels of Evidence Guidelines.21 As defined by Bernstein et al, basic science and biomechanical articles were classified as level V evidence.1 The highest LoE was recorded for each question that cited at least 1 primary journal article. For questions with more than 1 primary journal article in the references, the article with the highest LoE was recorded.
To compare “higher” and “lower” LoE, we defined higher LoE as level I, II, or III studies, and lower LoE were defined as levels IV and V, similar to the methodology used by Bernstein et al.1 “Source ages” were determined for each provided reference, which we calculated by subtracting the year of the referenced publication from the year that the OITE was administered.
Using the taxonomic classification proposed by Buckwalter et al, questions were classified as type 1 (T1), type 2 (T2), and type 3 (T3).2 T1 questions involve basic recall of information. T2 questions require examinees to determine an intraoperative problem, interpret an imaging study, or make a diagnosis. T3 questions are the most complex and require the examinee to make management decisions based on clinical vignettes.
Descriptive statistics were utilized to rank the most frequently cited references and to examine the mean age of the references used during each time period. We used chi-square testing to compare the percentages of questions in each time period that referenced at least 1 of each source type, referenced high LoE, and fit each category of the classification system proposed by Buckwalter et al. All statistical analysis was performed using SAS software (SAS 9.3, Cary, North Carolina). Differences of P < .05 were considered statistically significant.
Results
A total of 129 questions containing 222 references met inclusion criteria. Five questions were omitted from the examination and were thus excluded from our analysis. In total, 76 questions were from the 1995-1997 OITE and 53 were from 2010-2012 OITE. The Journal of Hand Surgery American and European Volumes (JHS), the Journal of Bone and Joint Surgery American and British Volumes (JBJS), and the Journal of the American Academy of Orthopaedic Surgeons (JAAOS) were the most frequently cited journals overall. JHS remained the most frequently cited journal in both time periods (Table 1). As noted in Table 2, recent examinations were more likely to cite multiple sources, with 100% of questions on the 2010-2012 citing 2 or more sources (P < .0001). Recent examinations were also more likely to cite 2 or more primary journal articles and 2 or more journal review articles compared with the 1995-1997 examinations (P = .002, P = .001, respectively). The mean age of all sources was 8.6 years (Table 3). The 2010-2012 OITE did not cite more recent sources compared with the 1995-1997 OITE when comparing all source types, primary journal articles only, or journal review articles only (P = 0.32, P = 0.64, P = 0.31, respectively).
Table 1.
Most Frequently Cited Journals, Overall and by Time Period.
| Rank | Overall (169 journal citations) | 1995-1997 (67 journal citations) | 2010-2012 (102 journal citations) |
|---|---|---|---|
| 1 | JHS (40%) | JHS (51%) | JHS (33%) |
| 2 | JBJS (11%) | JBJS (19%) | JAAOS (14%) |
| 3 | JAAOS (9%) | Hand Clinics (9%) | JBJS (7%) |
| 4 | Hand Clinics (7%) | CORR (6%) | Hand Clinics (7%) |
| 5 | CORR (5%) | 1 | JHS EUR (5%) |
Note. In 1995-1997, there were 10 journals that had one citation each. JHS, Journal of Hand Surgery; JBJS, Journal of Bone and Joint Surgery; JAAOS, Journal of the American Academy of Orthopaedic Surgeons; CORR, Clinical Orthopaedics and Related Research; JHS EUR, Journal of Hand Surgery European Volume.
Table 2.
Baseline Characteristics of the OITE Hand Questions, Overall and by Time Period.
| Overall | 1995-1997 | 2010-2012 | P value | |
|---|---|---|---|---|
| Total No. of questions examined, n | 129 | 76 | 53 | — |
| Buckwalter classification | .02 | |||
| T1, n (%) | 44 (34) | 23 (30) | 21 (40) | |
| T2, n (%) | 28 (22) | 23 (30) | 5 (9) | |
| T3, n (%) | 57 (44) | 30 (40) | 27 (51) | |
| No. of references per question | <.0001 | |||
| 1, n (%) | 48 (37) | 48 (63) | 0 (0) | |
| 2, n (%) | 69 (53) | 26 (34) | 43 (81) | |
| 3, n (%) | 12 (9) | 2 (3) | 10 (19) | |
| Types of reference per question | ||||
| At least 1 primary journal article, n (%) | 76 (59) | 40 (53) | 36 (58) | .08 |
| 2 or more primary journal articles, n (%) | 31 (24) | 11 (14) | 20 (38) | .002 |
| At least 1 journal review article, n (%) | 42 (33) | 13 (17) | 29 (55) | <.0001 |
| 2 or more journal review articles, n (%) | 13 (10) | 2 (3) | 11 (21) | .001 |
| At least 1 textbook, n (%) | 45 (35) | 33 (43) | 12 (23) | .01 |
| 2 or more textbooks, n (%) | 3 (2) | 2 (3) | 1 (2) | >.99 |
| At least 1 instructional course lecture, n (%) | 5 (4) | 4 (5) | 1 (2) | .33 |
| 2 or more instructional course lectures, n (%) | 0 (0) | 0 (0) | 0 (0) | — |
Note. OITE, Orthopaedic In-Training Examination.
Table 3.
Age of Sources Used Relative to the Year of the OITE.
| Overall | 1995-1997 | 2010-2012 | P value | |
|---|---|---|---|---|
| All sources | ||||
| Total No. of sources, n | 222 | 106 | 116 | — |
| Mean (SD) age of all sources, in years | 8.6 (9.3) | 9.3 (11.0) | 8.0 (7.4) | .32 |
| Primary journal articles only | ||||
| Total No. of primary journal articles, n | 112 | 52 | 60 | — |
| Mean (SD) age of all primary journal article sources, in years | 10.0 (10.7) | 10.6 (12.6) | 9.6 (8.8) | .64 |
| Journal review articles only | ||||
| Total No. of journal review articles, n | 57 | 15 | 42 | — |
| Mean (SD) age of all journal review article sources, in years | 7.7 (6.2) | 9.4 (8.1) | 7.1 (5.4) | .31 |
| Textbooks only | ||||
| Total No. of textbook sources, n | 48 | 35 | 13 | — |
| Mean (SD) age of all textbook sources, in years | 7.0 (8.8) | 8.1 (10.0) | 4.1 (2.9) | .03 |
Note. OITE, Orthopaedic In-Training Examination.
With respect to the question taxonomy for the hand surgery content domain on the OITE, the change in the distribution of the Buckwalter classification was statistically significant (P = .02). A total of 51% of hand questions on the 2010-2012 OITE were classified as T3 compared with 44% on the 1995-1997 examinations. The percentage of T2 decreased as well, comprising only 9% of hand questions of the 2010-2012 OITE (Table 2).
There was a statically significant increase in the LoE used to create hand questions on the 2010-2012 OITE compared with the 1995-1997 examinations. These results are outlined in Table 4. A total of 21% of questions on the 2010-2012 OITE cited level I to III evidence (“higher” LoE) compared with 7% on the 1995-1997 OITE (P = .016). When independently analyzing primary journal articles used as references, 25% of the 2010-2012 articles cited level I to III evidence compared with 10% of the 1995-1997 articles.
Table 4.
Comparison of the LoE Questions and References in Each Time Period.
| 1995-1997 OITE (n = 76) | 2010-2012 OITE (n = 53) | P value | |
|---|---|---|---|
| Questions in each time period that cited level I to III evidence, n (%) | .016 | ||
| Yes | 5 (7) | 11 (21) | |
| No | 35 (93) | 25 (79) | |
| 1995-1997 OITE (n = 52) | 2010-2012 OITE (n = 60) | P value | |
| Primary journal articles cited in each time period that contained level I to III evidence, n (%) | .03 | ||
| Yes | 5 (10) | 15 (25) | |
| No | 47 (90) | 45 (75) | |
Note. LoE, level of evidence; OITE, Orthopaedic In-Training Examination.
Discussion
The AAOS administers the OITE yearly to orthopedic surgery residents.11 Although it is not a validated examination, OITE performance has been correlated with ABOS Part 1 Examination performance and thus is an important component of resident education.8 During the past 10 years, most peer-reviewed orthopedic journals have published the LoE for original clinical articles,7,18,22 and many journals have made an effort to improve the LoE for published articles.5,16 Few studies in the literature have specifically addressed the hand surgery content domain on the OITE or LoE on the examination.1,4,6 We performed an analysis of the reference characteristics, question taxonomy, and LoE for primary journal articles used to create hand content domain questions over a 15-year period.
There are some notable limitations of our study. We chose to include only questions in the hand surgery content domain. Other domains such as musculoskeletal trauma and basic science frequently test hand surgery concepts. In choosing to analyze only hand surgery questions, it is possible that our results may not be generalizable to other content domains. A recent analysis of the musculoskeletal trauma content domain has been completed and found a similar increase in LoE; however, further study of other subspecialties needs to be completed.4 In addition, it is possible that the authors of the OITE questions cited articles with lower LoE or textbooks when articles with higher LoE existed for a given topic.
There appears to be a benefit in reviewing general orthopedic journals in addition to subspecialty journals for OITE and ABOS Part I preparation. JBJS, JAAOS, and Clinical Orthopaedics and Related Research (CORR) were all within the top 5 most frequently cited journals overall for both time periods. These findings have been echoed by other authors. In their study of OITE questions involving the hand or forearm from 2002 to 2006, Marker et al noted that the most frequently referenced journals were JBJS (American Volume), JBJS (British Volume), and CORR.12 Our data suggest that structured residency journal reviews of general and subspecialty publications may be beneficial in preparation for the OITE and ABOS Part 1 Board Examination, particularly those published within the past decade as the mean source age for references on the 2010-2012 OITE was 8 years. Although many factors contribute to success on written examinations, Miyamoto et al noted a correlation between successful OITE performance and frequent review of current orthopedic journals.13
Within the hand surgery content domain, there appears to be a shift toward clinical management questions as 51% questions on the 2010-2012 OITE were classified as T3. Previous authors, in analyzing subspecialty questions prior to 2010, noted that basic recall questions (T1) were the most common for the shoulder and elbow, sports medicine, basic science, and adult reconstruction content domains.3,14,17 Further study would be required to determine whether this trend toward management-based questions will continue or whether it is applicable to other subspecialty areas.
The LoE for questions within the hand surgery content domain on the OITE have increased over a 15-year period. Although the majority of primary journal articles cited on the examination were level IV or V studies, the improving LoE appears to mimic the increasing LoE found in general and subspecialty orthopedic journals. To et al analyzed the 50 most cited hand surgery articles since 1990 and found that 38% were level IV, 20% were level I, 16% were level III, and 4% were level II.19 In contrast, Lefaivre et al found no randomized, controlled trials within the 100 most frequently cited articles in orthopedics with 55% level IV studies.9
It appears that the hand surgery literature has shifted toward publishing studies with improved LoE, and the OITE is reflecting this improvement. Despite an overall improvement in LoE within the hand literature, there remains a relative lack of level I studies. Although there was an increase in the number of level I publications from 2.8% in 1991 to 3.9% from 2008 to 2011, there remains a paucity of randomized, controlled trials in the hand surgery literature.15 An examination consisting only of questions derived from level I studies would not adequately reflect the sum of hand surgery knowledge: Elements of clinical practice rely on basic science, an understanding of biomechanics, and expert opinion. For example, of the 29 recommendations in the 2010 AAOS Clinical Practice Guidelines for the treatment of distal radius fractures, none contains a “strong” recommendation grade meaning that no recommendation has more than 1 level I study with consistent findings.10 As it has been demonstrated that level 1 studies have a quicker route to publication, it is likely that LoE will continue to improve in peer-reviewed orthopedic journals.20
When reviewing journals, the presence of higher LoE studies may warrant closer scrutiny as it is becoming more likely that these studies may be referenced on the OITE. In addition, our results can be used as a guide to help prepare orthopedic residents for the OITE and suggest that reviewing both general and subspecialty journals may be beneficial for examination preparation.
Footnotes
Ethical Approval: This study was approved by our institutional review board.
Statement of Human and Animal Rights: This article does not contain any studies with human or animal subjects.
Statement of Informed Consent: This study did not include human subjects, thus no informed consent was obtained.
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
References
- 1. Bernstein J, Kenniston JA, Nydick JA, Zgonis MH, Beredjiklian PK. Levels of evidence are low for clinical management questions on the orthopaedic in-training examination. J Bone Joint Surg Am. 2010;92:508-511. [DOI] [PubMed] [Google Scholar]
- 2. Buckwalter JA, Schumacher R, Albright JP, Cooper RR. Use of an educational taxonomy for evaluation of cognitive performance. J Med Educ. 1981;56:115-121. [DOI] [PubMed] [Google Scholar]
- 3. Cross MB, Osbahr DC, Nam D, Reinhardt KR, Bostrom MP, Dines JS. An analysis of the hip and knee reconstruction section of the orthopaedic in-training examination. Orthopedics. 2011;34(9):e550-e555. [DOI] [PubMed] [Google Scholar]
- 4. Grandizio LC, Huston JC, Shim SS, Parenti JM, Graham J, Klena JC. Levels of evidence have increased for musculoskeletal trauma questions on the orthopaedic in-training examination. J Surg Educ. 2014;72(2):258-263. [DOI] [PubMed] [Google Scholar]
- 5. Hanzlik S, Mahabir RC, Baynosa RC, Khiabani KT. Levels of evidence in research published in The Journal of Bone and Joint Surgery (American Volume) over the last thirty years. J Bone Joint Surg Am. 2009;91:425-428. [DOI] [PubMed] [Google Scholar]
- 6. Haughom BD, Goldstein Z, Hellman MD, Yi PH, Frank RM, Levine BR. An analysis of references used for the orthopaedic in-training examination: what are their levels of evidence and journal impact factors? Clin Orthop Relat Res. 2014;472:4024-4032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Hentz RV, Meals RA, Stern P, Manske PR. Levels of evidence and the journal of hand surgery. J Hand Surg Am. 2005;30(5):891-892. [DOI] [PubMed] [Google Scholar]
- 8. Herndon JH, Allan BJ, Dyer G, Jawa A, Zurakowski D. Predictors of success on the American Board of Orthopaedic Surgery examination. Clin Orthop Relat Res. 2009;467:2436-2445. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Lefaivre KA, Shadgan B, O’Brien PJ. 100 most cited articles in orthopaedic surgery. Clin Orthop Relat Res. 2011;469(5):1487-1497. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Lichtman DM, Randipsingh RB, Boyer MI, et al. Treatment of distal radius fractures. J Am Acad Orthop Surg. 2010;18(3):180-189. [DOI] [PubMed] [Google Scholar]
- 11. Mankin HJ. The orthopaedic in-training examination (OITE). Clin Orthop Relat Res. 1971;75:108-116. [DOI] [PubMed] [Google Scholar]
- 12. Marker DR, Mont MA, McGrath MS, Frassica FJ, LaPorte DM. Current hand surgery literature as an educational tool for the orthopaedic in-training examination. J Bone Joint Surg Am. 2009;91(1):236-240. [DOI] [PubMed] [Google Scholar]
- 13. Miyamoto RG, Klein GR, Walsh M, Zuckerman JD. Orthopedic surgery residents’ study habits and performance on the orthopedic in-training examination. Am J Orthop. 2007;36(12):e185-188. [PubMed] [Google Scholar]
- 14. Osbahr DC, Cross MB, Bedi A, et al. Orthopaedic in-training examination: an analysis of the sports medicine section. Am J Sports Med. 2011;39(3):532-537. [DOI] [PubMed] [Google Scholar]
- 15. Post SF, Selles RW, McGrouther DA, et al. Levels of evidence and quality of randomized controlled trials in hand and wrist surgery: an analysis of two major hand surgery journals [editorial]. J Hand Surg Eur. 2014;39(8):900-902. [DOI] [PubMed] [Google Scholar]
- 16. Razi Z, Abbassian A, Cro S, et al. Levels of evidence in foot and ankle surgery literature: progress from 2000 to 2010? J Bone Joint Surg Am. 2012;94:e1121-e1110. [DOI] [PubMed] [Google Scholar]
- 17. Sheibani-Rad S, Arnoczky SP, Walter NE. Analysis of the basic science section of the orthopaedic in-training examination. Orthopedics. 2012;35(8):1251-1255. [DOI] [PubMed] [Google Scholar]
- 18. Slobogean GP, Bhandari M. Introducing levels of evidence to the Journal of Orthopaedic Trauma: implementation and future directions. J Orthop Trauma. 2012;26:127-128. [DOI] [PubMed] [Google Scholar]
- 19. To P, Atkinson CT, Lee DH, Pappas ND. The most cited articles in hand surgery over the past 20-plus years: a modern-day reading list. J Hand Surg Am. 2013;38(5):983-987. [DOI] [PubMed] [Google Scholar]
- 20. Voleti PB, Donegan DJ, Baldwin KD, Lee GC. Level of evidence of presentations at American Academy of Orthopaedic Surgeons annual meetings. J Bone Joint Surg Am. 2012;94:e50-e51. [DOI] [PubMed] [Google Scholar]
- 21. Wright JG. Levels of evidence and grades of recommendation (AAOS website). http://www2.aaos.org/bulletin/apr05/fline9.asp. Accessed April 3, 2014.
- 22. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85(1):1-3. [PubMed] [Google Scholar]