Abstract
Background
Although the references recommended for the Orthopaedic In-Training Examination (OITE) have been evaluated in certain subspecialty domains, suggested reference level of evidence (LOE), impact factor, and citation age have not been evaluated comprehensively to our knowledge.
Questions/purposes
We present an analysis of all references cited in the OITE recommended readings for each test question including the duration of time between their initial publication and their use in the OITE, which we defined as citation age, LOE, and the impact factor of the journals referenced.
Methods
We evaluated all references for the 2010 to 2012 OITE administrations (three examinations; 825 questions total). Publication characteristics, including citation age, were noted. The LOE for each journal article and the impact factor of each journal were determined; differences in LOE and impact factor were compared between test sections. A total of 1817 references were cited in the 825 questions we evaluated; this denominator was used in all calculations that follow.
Results
The recommended reading references included 1337 journal article references (74%), 469 text references (26%), and 11 multimedia sources (0.6%; eg, websites, instructional DVDs). The three most commonly recommended journals were general orthopaedic journals, TheJournal of Bone and Joint Surgery (American Volume), Journal of American Academy of Orthopaedic Surgeons, and Clinical Orthopaedics and Related Research®. The majority (72.2%) of the cited journal references were published within 10 years of the test date, with a mean ± SD citation age of 8.3 ± 7.4 years. The majority of the cited journal articles were Levels IV and V evidence (mean, 4.16 ± 1.1). The Spine section had higher LOE (3.74; p < 0.001), although the practical relevance of such a difference is questionable, as all but two sections’ LOE rounded to Level IV evidence. The Spine and Basic Science sections were published in journals with a larger mean impact factor (Basic Science, 7.16 ± 12.67; Spine, 5.73 ± 12.08; p < 0.001).
Conclusions
Our data show that the majority of the recommended readings for the OITE stem from higher impact general orthopaedic and major subspecialty journals. Furthermore the observed mean LOE of the recommended readings shows a preponderance of Levels IV and V research. These data may suggest that test-takers may find benefit in the review of high-level general orthopaedic journals, and review articles in particular while preparing for the OITE, although further study is necessary to determine optimal test preparation strategies. Finally, our study provides a baseline analysis of the study designs of OITE recommended references, and may provide insight for educators designing resident educational curricula.
Electronic supplementary material
The online version of this article (doi:10.1007/s11999-014-3895-0) contains supplementary material, which is available to authorized users.
Introduction
The Orthopaedic In-Training Examination (OITE), first administered in 1963 by the American Academy of Orthopaedic Surgeons (AAOS), has become a cornerstone in the educational evaluation of orthopaedic surgery residents. Its initial intentions were to help orthopaedic surgery residents and residency programs evaluate the effectiveness of their education programs, as measured by residents’ knowledge relative to their peers [9]. Interest in improving residency education is increasing, with an expansion in the number of studies evaluating the topic [10] and introduction of new initiatives by the Accreditation Council for Graduate Medical Education, such as mandatory orthopaedic skills training during the intern year.
OITE performance correlates with successful completion of the American Board of Orthopaedic Surgery (ABOS) Part I Exam [5, 14]. One investigation examined the effectiveness of online and review texts as an ideal means of preparing for these tests [8]. Understanding the types of references recommended for review by the OITE and what journals or texts these references come from, however, may provide insight for educators to design effective and optimal educational curricula (with the goal being the ability to teach residents orthopaedics and at the same time cover the relevant topics targeted on the OITE and boards). Although one study evaluated the levels of evidence (LOE) of the references cited by the “clinical management questions” on the OITE [3], the overall LOE of the references cited by OITE questions, to our knowledge, has not been evaluated. Likewise, an analysis to determine whether the references are reasonably current, and an analysis of the quality of the journals from which the references are drawn using an external benchmark, such as the impact factor [15], to our knowledge, has not been performed.
We therefore (1) provide a comprehensive description of the references most frequently recommended by the OITE including the citation age of the references cited; (2) evaluate the LOE of all journal references cited, with a breakdown by subspecialty section; and (3) provide an analysis of the impact factor of the journal references cited.
Materials and Methods
The AAOS provides a test key for each yearly administration of the OITE, which indicates the subspecialty domain and recommended readings for each question on the test. The subspecialty domains include Basic Science and Tumor, Foot and Ankle, Hand, Hip and Knee Reconstruction, Medically Related Issues, Musculoskeletal Trauma, Orthopaedic Diseases, Oncology, Pediatric Orthopaedics, Rehabilitation, Shoulder and Elbow, Spine, and Sports Medicine. During the study period (2010–2012), three examinations were administered, for a total of 825 questions.
After obtaining institutional review board approval, we reviewed the test keys for the 2010 to 2012 administrations of the OITE for the recommended readings. There were some changes in the subject domains during the test years examined. Specifically, the 2010 and 2011 examinations included a Rehabilitation section and a Basic Science and Tumor section, while the 2012 examination replaced the Basic Science and Tumor section with a Basic Science and Orthopaedic Diseases section, eliminated the Rehabilitation section, and added a dedicated Oncology section. In light of these changes, we decided to analyze sections according to their original designation (eg, the Basic Science and Tumor questions were analyzed separately from the Basic Science and Orthopaedic Diseases questions).
Each of the recommended reading references was obtained, and the following information was gathered: test year, question number, subspecialty domain, citation age of publication (integer difference between test year and year of each reference’s publication), LOE, and impact factor. LOE and impact factor were applicable only to the journal references. The LOE was ascertained by one reviewer (BDH) blinded to the test year and question. The LOE was determined using well-described metrics (eg, Level I = well done randomized control trials and systematic reviews of Level I research; Level V = nonsystematic reviews and expert opinion) [3, 17]. In the situation where an editorial staff reported the LOE in the manuscript, it was used if the authors agreed with their determination; there were no instances where a discrepancy occurred. In any situation where the LOE was unclear, three reviewers (BDH, PHY, MDH) reviewed the reference in question to determine the appropriate LOE.
Impact factor is a calculated metric, produced by the Institute for Scientific Information (ISI) and Thompson Reuters (an international media corporation), which determines the average number of citations a journal’s papers receive during a 2-year (or 5-year) period. This metric does not apply to text or multimedia sources. The calculation is performed by dividing the number of times articles published in a particular journal, in the preceding 2 years, were cited in indexed journals by the total number of citations published in that journal during the same period. Although this metric provides an approximate estimate for the relative importance of a particular journal in its subspecialty field, it comes with certain limitations. The measurement is more relevant as a metric of an overall journal as opposed to an individual specific article, as it aggregates the citations of all articles of a particular journal [15]. The calculation also may not be entirely accurate, as it assumes an arithmetic mean as opposed to a geometric or exponential increase in citation volume [1]. Furthermore, editorial boards can adopt policies (eg, coercive citation) to artificially inflate their journal’s impact factor calculation and subsequent perceived importance [16]. The 2-year impact factor was used in our analysis, which is a measure that reflects the average number of recent citations of a journal during a 2-year period [15]. To minimize the inherent temporal variability with this metric, the impact factor was obtained at one time (April 1, 2013) for all journals using the ISI web interface [15].
The recommended readings for all questions on the 2010 to 2012 OITE were included in our analysis. This includes questions removed by the AAOS for scoring consideration after the exam for various reasons, such as ambiguous answers or unclear wording. We made the decision to include these references as we believed the selected references provided valuable information regarding the quality and spectrum of the references provided by the AAOS.
Means and SDs were calculated for continuous variables. Comparisons between the LOE, impact factor, and citation age were drawn between subspecialty domains across test years and in subspecialty domains in the same test year via ANOVA. Statistical significance was set at a level of 0.05 or less. All statistical analyses were performed using SPSS® software (Version 18; SPSS Inc, Chicago, IL, USA).
Results
There were 1817 recommended reading references, including 1337 journal article references (75%), 469 text references (26%), and 11 multimedia sources (0.8%; eg, websites, instructional DVDs). Of the 1337 journal articles, 99 articles (7%) were cited more than once. The three most commonly cited journals were TheJournal of Bone and Joint Surgery (American Volume; JBJS), Journal of American Academy of Orthopaedic Surgeons (JAAOS), and Clinical Orthopaedics and Related Research® (CORR®) (Table 1). The two journals from which articles were repeated most often included JBJS with 18 repeated articles, and JAAOS with 28 repeated articles (Fig. 1). Moreover, these core journals were commonly cited across subspecialty domains (Table 2). The three most commonly cited text references included: Orthopaedic Basic Science: Foundations of Clinical Practice, AAOS Comprehensive Orthopaedic Review, and Skeletal Trauma: Basic Science, Management, and Reconstruction (Fig. 2). The mean citation age of the selected references was 8.3 ± 7.4 years across the study period; references did not show differences regarding citation age across the study period (Table 3). The calculated reference citation age ranged between 0 to 55 years, with the oldest referenced study published in JBJS (1955). Of the suggested references, 7.8% were published in the year of or the year before the test administration, while 45.9% of the references were published within 5 years of the test year, and 27.8% were published greater than 10 years from the test year.
Table 1.
The most common journal and text references cited on the 2010 to 2012 OITE
| Rank | Title | Number of citations |
|---|---|---|
| Journals | ||
| 1 | Journal of Bone Joint Surgery (American Volume) | 225 |
| 2 | Journal of the American Academy of Orthopaedic Surgeons | 186 |
| 3 | Clinical Orthopaedics and Related Research ® | 86 |
| 4 | Journal of Orthopaedic Trauma | 75 |
| 5 | American Journal of Sports Medicine | 64 |
| 6 | Spine | 49 |
| 7 | Journal of Hand Surgery (American Volume) | 47 |
| 8 | Journal of Bone Joint Surgery (British Volume) | 40 |
| 9 | Foot and Ankle International | 39 |
| 10 | Journal of Pediatric Orthopaedics | 38 |
| Textbooks | ||
| 1 | Orthopaedic Basic Science: Foundations of Clinical Practice | 57 |
| 2 | Instructional Course Lecture | 40 |
| 2 | AAOS Comprehensive Orthopaedic Review | 21 |
| 3 | Skeletal Trauma: Basic Science Management, and Reconstruction | 19 |
| 4 | Lovell and Winter’s Pediatric Orthopaedics | 18 |
| 5 (tie) | Orthopaedic Knowledge Update 10 | 17 |
| 5 (tie) | Orthopaedic Knowledge Update: Musculoskeletal Tumors 2 | 17 |
OITE = Orthopaedic In-Training Examination.
Fig. 1.
A graph shows the most commonly cited recommended journal citations for the OITE (2010–2012). The two journals from which articles most often were repeated included The Journal of Bone and Joint Surgery (American Volume)(JBJS) with 18 repeated articles, and Journal of the American Academy of Orthopaedic Surgeons (JAAOS) with 28 repeated journal articles. CORR = Clinical Orthopaedics and Related Research ®; JOT = Journal of Orthopaedic Trauma; AJSM = American Journal of Sports Medicine; J Hand Surg Am = Journal of Hand Surgery (American); JBJS Br = Journal of Bone and Joint Surgery (British Volume); Foot Ankle Int = Foot and Ankle International; J Pediatr Orthop = Journal of Pediatric Orthopaedics; JSES = Journal of Shoulder and Elbow Surgery; J Arthroplasty = Journal of Arthroplasty; J Trauma = Journal of Trauma.
Table 2.
The three most commonly cited references (texts and journals) across subspecialty domains
| Subspecialty domain | Most commonly cited journals and textbooks | Number of references |
|---|---|---|
| Basic Science and Tumors | Orthopaedic Basic Science: Foundations of Clinical Practice | 36 |
| Journal of Bone and Joint Surgery (American Volume) | 11 | |
| AAOS Comprehensive Orthopaedic Review | 9 | |
| Basic Science and Orthopaedic Diseases | Orthopaedic Basic Science: Foundations of Clinical Practice | 15 |
| AAOS Comprehensive Orthopaedic Review | 10 | |
| Journal of Bone and Joint Surgery (American Volume) | 9 | |
| Foot and Ankle | Foot Ankle International | 29 |
| Journal of Bone and Joint Surgery (American Volume) | 15 | |
| Surgery of the Foot and Ankle | 13 | |
| Hand | Journal of Hand Surgery (American) | 34 |
| Journal of the American Academy of Orthopaedic Surgeons | 14 | |
| Hand Clinics | 7 | |
| Hip and Knee Reconstruction | Journal of Bone and Joint Surgery (American Volume) | 30 |
| Clinical Orthopaedics and Related Research ® | 27 | |
| Journal of the American Academy of Orthopaedic Surgeons | 25 | |
| Medically Related Issues | Journal of Bone and Joint Surgery (American Volume) | 7 |
| Journal of the American Academy of Orthopaedic Surgeons | 3 | |
| Orthopaedic Basic Science: Foundations of Clinical Practice | 3 | |
| Musculoskeletal Trauma | Journal of Orthopaedic Trauma | 60 |
| Journal of Bone and Joint Surgery (American Volume) | 46 | |
| Journal of the American Academy of Orthopaedic Surgeons | 38 | |
| Oncology | Journal of the American Academy of Orthopaedic Surgeons | 6 |
| Orthopaedic Knowledge Update: Musculoskeletal Tumors 2 | 4 | |
| Clinical Orthopaedics and Related Research ® | 3 | |
| Orthopaedic Diseases | Orthopaedic Knowledge Update: Musculoskeletal Tumors 2 | 9 |
| Orthopaedic Surgery Essentials, Oncology and Basic Science | 8 | |
| Journal of the American Academy of Orthopaedic Surgeons | 6 | |
| Pediatric Orthopaedics | Journal of Bone and Joint Surgery (American Volume) | 31 |
| Journal of the American Academy of Orthopaedic Surgeons | 27 | |
| Journal of Pediatric Orthopaedics | 26 | |
| Rehabilitation | American Journal of Sports Medicine | 6 |
| Journal of Bone and Joint Surgery (American Volume) | 5 | |
| Journal of Orthopaedic Trauma | 3 | |
| Shoulder and Elbow | Journal of Bone and Joint Surgery (American Volume) | 27 |
| Journal of Shoulder and Elbow Surgery | 18 | |
| Journal of the American Academy of Orthopaedic Surgeons | 10 | |
| Spine | Spine | 45 |
| Journal of the American Academy of Orthopaedic Surgeons | 19 | |
| Journal of Bone and Joint Surgery (American Volume) | 12 | |
| Sports Medicine | American Journal of Sports Medicine | 39 |
| Journal of Bone and Joint Surgery (American Volume) | 19 | |
| Arthroscopy | 10 |
Fig. 2.
A graph shows the most commonly cited text citations among the recommended readings for the OITE (2010–2012). The three most commonly cited text references included: Orthopaedic Basic Science: Foundations of Clinical Practice, AAOS Comprehensive Orthopaedic Review, and Skeletal Trauma: Basic Science, Management, and Reconstruction.
Table 3.
Characteristics of the 2010 to 2012 OITE recommended readings
| Subspecialty domain | Level of evidence | Impact factor | Citation age (years) |
|---|---|---|---|
| Basic Science and Tumor | 4.39 (1.22) | 7.16 (12.67)* | 7.83 (5.73) |
| Basic Science and Orthopaedic Diseases | 4.53(1.11) | 3.24 (1.82) | 7.00 (5.33) |
| Foot and Ankle | 4.15 (0.97) | 2.46 (3.23) | 9.55 (7.69) |
| Hand | 4.43 (0.85) | 2.09 (1.68) | 8.34 (7.61) |
| Hip and Knee Reconstruction | 4.04 (1.17) | 2.57 (0.57) | 7.10 (5.90) |
| Medically Related Issues | 4.06 (1.10) | 3.75 (5.08) | 8.82 (9.96) |
| Musculoskeletal Trauma | 4.17 (1.11) | 2.42 (0.83) | 8.78 (9.96) |
| Oncology | 4.41 (0.87) | 3.68 (6.47) | 8.69 (6.09) |
| Orthopaedic Diseases | 4.03 (1.46) | 3.67 (5.97) | 5.97 (6.55) |
| Pediatric Orthopaedics | 4.17 (1.05) | 3.17 (6.43) | 9.25 (8.84) |
| Rehabilitation | 4.00 (1.26) | 2.91 (0.96) | 11.40 (10.50) |
| Shoulder and Elbow | 4.04 (1.02) | 2.83 (0.66) | 8.76 (6.99) |
| Spine | 3.74 (1.22)* | 5.73 (12.08)* | 7.23 (7.41) |
| Sports Medicine | 4.31 (0.94) | 3.29 (2.45) | 6.73 (5.40) |
| Overall mean | 4.16 (1.10) | 3.23 (5.53) | 8.28 (7.39) |
Values are expressed as mean, with SD in parentheses; *p < 0.005; OITE = Orthopaedic In-Training Examination.
Regarding the journal articles referenced, the mean LOE of the selected references was 4.6 (Table 4). Regarding frequency of LOE, Level I research represented 3.0%, Level II 8.0%, Level III 11.5%, Level IV 25.2%, and Level V 52.5% of the studies referenced during the 3-year period. The mean LOE each year was consistent throughout the studied period (Fig. 3). In contrast to the other subspecialty domains, the Spine domain had a higher average LOE (LOE, 3.74; p < 0.001) (Table 3) (Appendix 1. Supplemental material is available with the online version of CORR®), with a small measured effect size (F-ratio, 3.557). Specifically, the Spine section had higher LOE compared with the Basic Science and Tumor, Hand, Musculoskeletal Trauma, Pediatric Orthopedics, and Sports sections, respectively.
Table 4.
Percentage breakdown of the recommended journal article LOE on the 2010 to 2012 OITE
| LOE | 2010 (%) | 2011 (%) | 2012 (%) | 2010–2012 (%) |
|---|---|---|---|---|
| I | 3.8 | 2.9 | 2.4 | 3.0 |
| II | 11.3 | 5.5 | 7.8 | 8.0 |
| III | 9.6 | 12.0 | 12.7 | 11.5 |
| IV | 21.4 | 25.4 | 28.2 | 25.2 |
| V | 53.9 | 54.2 | 48.9 | 52.3 |
LOE = level of evidence; OITE = Orthopaedic In-Training Examination.
Fig. 3.
A graph shows the number of citations per year stratified by journal article LOE. The mean LOE each year was consistent throughout the studied period.
Similarly, the impact factor across the subspecialty domains did not differ with the exception of the Spine and Basic Science references, which were more likely to be published in higher-impact journals (Basic Science impact factor, 7.16 ± 12. 67; Spine impact factor, 5.73 ± 12.08; p < 0.001) (Appendix 2. Supplemental material is available with the online version of CORR®) (Fig. 4), although the observed effect size was small (F-Ratio, 5.917). Specifically, the journals referenced in the Spine section had a higher impact factor compared with the journals referenced in the Foot and Ankle, Hand, Hip and Knee Reconstruction, Musculoskeletal Trauma, Rehabilitation, Shoulder and Elbow, and Sports sections (p < 0.05 for all comparisons), while the journals referenced in the Basic Science and Tumor section had a higher impact factor compared with the journals referenced in the Foot and Ankle, Hand, Hip and Knee Reconstruction, Musculoskeletal Trauma, and Shoulder and Elbow sections (p < 0.05 for all comparisons).
Fig. 4.
A graph shows the mean LOE, impact factor, and citation age of the journal articles according to subspecialty domain cited by the OITE (2010–2012). Statistical significance was observed in the Spine section (LOE and impact factor) and the Basic Science section (impact factor) (*p < 0.001). No difference was observed with respect to article citation age.
Discussion
The OITE was created to serve as a benchmark for resident performance and as a tool to gauge an individual residency’s educational program [9]. Given its ubiquity in resident education and its prognostic relationship with the ABOS Part I Exam [14], the OITE has become an important force in shaping orthopaedic curricula across the country. Accordingly, some studies have been conducted evaluating orthopaedic education, with specific emphasis on the OITE [3, 10]. As clinicians and educators increasingly emphasize evidence-based care, it is unknown if the OITE has followed suit. Understanding the types and sources of references being recommended for OITE preparation might help educators to design appropriate curricula and resident reading lists. To our knowledge, no prior study has performed a comprehensive assessment of the LOE, citation age, or impact factor of the references and journals suggested by the test writers of the OITE. We therefore performed an analysis of the references recommended for the OITE and an appraisal of the LOE of all suggested references, the impact factors of the journals referenced, and the mean citation age of the references cited during our study period.
Our study has several limitations. First, we were unable to provide any correlation between the LOEs of references and impact factors of journals referenced and the performance on the OITE, such as percentage of test-takers correctly answering a test question. Second, because of changes to the Basic Science, Tumor, Rehabilitation, and Orthopaedic Diseases sections, certain domains provided fewer citations for evaluation, which may skew the results of our LOE and impact factor analyses. It also was difficult to assign a given question to a given subspecialty, as in many cases the citation applied to both. Third, the LOE is not necessarily an indicator of a study’s quality, since high-quality Level IV case series may coexist with Level I studies which are of low quality owing to factors other than those used to assign a LOE, such as sample size, statistical analysis, or loss to followup. Finally, while LOE provides a general framework of a study’s strength, it may fall short in a well-done nonsystematic review. Another limitation of our study is that focus on LOE and impact factor diverts attention from the overall focus of the OITE, which is to assess trainee knowledge. The heavy reliance on the JAAOS, which tends to focus on nonsystematic review articles (Level V evidence), may reduce the overall LOE in the OITE as an artifact. We also are unable to comment on the differences in LOE between article types (eg, original research, review article, meta-analysis); this analysis is outside the scope of our study, and remains a topic we plan to study in subsequent investigations.
The importance of general orthopaedic journals is underscored by our results and is consistent with previous reports that suggest that these publications are useful for OITE preparation [10, 11]. Regarding the journals referenced, there have not been significant changes in recent years, as the three most commonly cited journals remained unchanged (JBJS, CORR®, and JAAOS). The prevalence of JAAOS articles being referenced may explain the relatively high prevalence of Level V literature cited, as they primarily publish nonsystematic reviews. The most commonly cited texts included classic textbooks such as Skeletal Trauma, and Lovell and Winter’s Pediatric Orthopaedics. We separated Orthopaedic Knowledge Update into subspecialty sections, which likely accounts for its lesser prevalence in our list. Nevertheless, our list of the most commonly referenced sources confirms that the most well-known and widely read journals and textbooks are well represented on the OITE, consistent with the sources that have been anecdotally most emphasized in our resident training program. That more than ¼ of the referenced articles have a citation age greater than 10 years probably reflects the inclusion of papers that are likely to be included on the OITE for several years beyond their initial publication date. In contrast, nearly ½ of all references were published within 5 years of the OITE administration year, indicating that questions tend to be updated regularly.
Our review of the 2010 to 2012 examinations shows that most of the studies cited by the OITE are Levels IV and V papers. The high percentage of Level V citations may suggest the relative importance of reviewing contemporary review articles in preparation for the OITE, consistent with previous assessments of the OITE [3]. Although the Spine section was shown to have higher average LOE, the practical relevance of this finding is questionable, as the LOE for all but two of the sections (Basic Science & Tumor, and Basic Science and Orthopaedic Diseases) numerically round to Level IV work, and is consistent with our overall results. While the interpretation of LOE has been shown to have questionable reproducibility [12] and the LOE is not the ultimate determinant of a paper’s quality, it provides a framework for the reader to evaluate the potential bias present in a study’s design. In general, Levels I and II evidence is preferable, however, such randomized and/or nonrandomized prospective studies are not always possible for various reasons and situations (eg, a randomized controlled trial of a very rare congenital orthopaedic condition). Nevertheless, our findings show a similar proportion of Levels I and II literature to the reported values reported by Cunningham et al. [4] who observed a trend of increasing LOE between 2000 and 2010. Given the preponderance of Level V evidence, a large proportion of which was comprised of JAAOS review articles, devoting a larger proportion of an orthopaedic resident’s educational curriculum and reading list toward these kinds of articles may be a more efficient and efficacious way of preparing residents for the OITE and general orthopaedic knowledge.
Orthopaedics as a field is recognizing the need for higher-quality evidence, with some studies drawing light to the impact factor of journals [6, 7, 13]. Although the impact factor of orthopaedic journals remains low relative to other specialty journals, this is not surprising given the relatively small size of the field compared with specialties such as internal medicine. For the OITE, recognizing where the references suggested are coming from (ie, high- versus low-impact journals) is important to evaluate what information orthopaedic residents are expected to know, and provide a foundation and some structure to OITE preparation. Our data suggest that the literature for the Basic Science and Spine sections comes from higher-impact journals. Amiri et al. [2] reported a direct correlation with impact factor and higher LOE in the spine literature. Our data, particularly in the Spine section, confirmed this trend. However, because the majority of basic science literature is Level V, this correlation does not hold in this section.
Our data suggest a role for the review of high-level general orthopaedic journals, and review articles in particular, in preparation for the OITE, given the preponderance of such articles in the suggested references on the OITE. However, focusing on the review of such articles may be less efficient for the busy orthopaedic resident preparing for the OITE than other strategies, such as completing practice questions from previous OITE examinations, which we believe may be a fruitful topic for future study. We hope our data will serve as a baseline for future studies to evaluate the improvement in quality of the references cited on the OITE. Perhaps more importantly, however, as residents prepare and as educators advise their residents for their preparations for future OITE exams, we hope that our findings will help guide study to be more efficient and high-yield.
Electronic supplementary material
Acknowledgments
We thank Phyllis Velez C-TAGME, Program Administrator, Orthopaedic Surgery Residency Program, Rush University, for her tireless efforts and assistance throughout this project, particularly in obtaining the OITE exam questions and recommended references.
Footnotes
Each author certifies that he or she, or a member of his or her immediate family, has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing, etc) that might pose a conflict of interest in connection with the submitted article.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
This work was performed at Rush University Medical Center, Chicago, IL, USA.
References
- 1.Adler R, Ewing J, Taylor P. Citation statistics: a report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS) Statist Sci. 2009;24:1–14. doi: 10.1214/09-STS285. [DOI] [Google Scholar]
- 2.Amiri AR, Kanesalingam K, Cro S, Casey AT. Level of evidence of clinical spinal research and its correlation with journal impact factor. Spine J. 2013;13:1148–1153. doi: 10.1016/j.spinee.2013.05.026. [DOI] [PubMed] [Google Scholar]
- 3.Bernstein J, Kenniston JA, Nydick JA, Zgonis MH, Beredjiklian PK. Levels of evidence are low for clinical management questions on the orthopaedic in-training examination. J Bone Joint Surg Am. 2010;92:508–511. doi: 10.2106/JBJS.I.00530. [DOI] [PubMed] [Google Scholar]
- 4.Cunningham BP, Harmsen S, Kweon C, Patterson J, Waldrop R, McLaren A, McLemore R. Have levels of evidence improved the quality of orthopaedic research? Clin Orthop Relat Res. 2013;471:3679–3686. doi: 10.1007/s11999-013-3159-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Dyrstad BW, Pope D, Milbrandt JC, Beck RT, Weinhoeft AL, Idusuyi OB. Predictive measures of a resident’s performance on written orthopaedic board scores. Iowa Orthop J. 2011;31:238–243. [PMC free article] [PubMed] [Google Scholar]
- 6.Hakkalamani S, Rawal A, Hennessy MS, Parkinson RW. The impact factor of seven orthopaedic journals: factors influencing it. J Bone Joint Surg Br. 2006;88:159–162. doi: 10.1302/0301-620X.88B2.16983. [DOI] [PubMed] [Google Scholar]
- 7.Kodumuri P, Ollivere B, Holley J, Moran CG. The impact factor of a journal is a poor measure of the clinical relevance of its papers. Bone Joint J. 2014;96:414–419. doi: 10.1302/0301-620X.96B3.32279. [DOI] [PubMed] [Google Scholar]
- 8.Krueger CA, Shakir I, Fuller BC. Prevalence of answers to orthopaedic in-training examination questions in 3 commonly used orthopedic review sources. Orthopedics. 2012;35:e1420–e1426. doi: 10.3928/01477447-20120822-32. [DOI] [PubMed] [Google Scholar]
- 9.Mankin HJ. The Orthopaedic In-Training Examination (OITE) Clin Orthop Relat Res. 1971;75:108–116. doi: 10.1097/00003086-197103000-00014. [DOI] [PubMed] [Google Scholar]
- 10.Marker DR, LaPorte DM, Seyler TM, Ulrich SD, McGrath MS, Frassica FJ, Mont MA. Orthopaedic journal publications and their role in the preparation for the orthopaedic in-training examination. J Bone Joint Surg Am. 2009;91(suppl 6):59–66. doi: 10.2106/JBJS.I.00559. [DOI] [PubMed] [Google Scholar]
- 11.Miyamoto RG, Jr, Klein GR, Walsh M, Zuckerman JD. Orthopedic surgery residents’ study habits and performance on the orthopedic in-training examination. Am J Orthop (Belle Mead NJ). 2007;36:E185–E188. [PubMed] [Google Scholar]
- 12.Schmidt AH, Zhao G, Turkelson C. Levels of evidence at the AAOS meeting: can authors rate their own submissions, and do other raters agree? J Bone Joint Surg Am. 2009;91:867–873. doi: 10.2106/JBJS.G.01233. [DOI] [PubMed] [Google Scholar]
- 13.Siebelt M, Siebelt T, Pilot P, Bloem RM, Bhandari M, Poolman RW. Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for impact factor and SCImago. BMC Musculoskelet Disord. 2010;11:4. doi: 10.1186/1471-2474-11-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Swanson D, Marsh JL, Hurwitz S, DeRosa GP, Holtzman K, Bucak SD, Baker A, Morrison C. Utility of AAOS OITE scores in predicting ABOS Part I outcomes: AAOS exhibit selection. J Bone Joint Surg Am. 2013;95:e84. doi: 10.2106/JBJS.L.00457. [DOI] [PubMed] [Google Scholar]
- 15.Web of Knowledge. Available at: webofknowledge.com. Accessed April 1, 2013.
- 16.Wilhite AW, Fong EA. Scientific publications: coercive citation in academic publishing. Science. 2012;335:542–543. doi: 10.1126/science.1212540. [DOI] [PubMed] [Google Scholar]
- 17.Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85:1–3. doi: 10.1302/0301-620X.85B1.14063. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.