Abstract
Background:
Popularization of systematic reviews has been met with controversy because of concerns that the primary literature for certain topics may not be suited for systematic review and meta-analysis.
Purpose:
To assess the rate of publication of systematic reviews based on their level of evidence (LOE) in influential orthopaedic sports medicine journals and commonly studied topics in sports medicine.
Study Design:
Systematic review.
Methods:
An electronic search was performed using the PubMed database of studies published from January 2010 to December 2020. The advanced search function was used to identify systematic reviews from the Journal of Shoulder and Elbow Surgery (JSES), American Journal of Sports Medicine (AJSM), Arthroscopy, British Journal of Sports Medicine (BJSM), Journal of Bone and Joint Surgery–American Volume (JBJS), and Sports Medicine (SM Auckland), as well as reviews of the most common areas of sports medicine research, including rotator cuff repair (RCR), shoulder instability (SI), anterior cruciate ligament reconstruction (ACLR), and meniscal repair. The LOE was assigned to each included study according to the Oxford Centre for Evidence-Based Medicine. Studies were grouped as LOE 1-2, LOE 3-5, and nonclinical systematic reviews. A negative binomial regression was used to determine the changes in publication rate over time.
Results:
A total of 2162 systematic reviews were included in this study. From 2010 to 2020, the rate of publication of LOE 3-5 systematic reviews increased significantly among most of the surveyed journals (AJSM, P < .0001; Arthroscopy, P = .01; BJSM, P < .0001; JSES, P < .0001; SM Auckland, P < .0001), with the exception of JBJS (P = .57). The rate of publication of LOE 1-2 systematic reviews increased in AJSM (P < .0001), Arthroscopy (P = .02), BJSM (P < .0001), and SM Auckland (P < .0001); however, no significant changes were seen in JBJS (P = .08) or JSES (P = .15). The publication rate of LOE 3-5 systematic reviews increased for all sports medicine topics surveyed (meniscal repair, P < .0001; RCR, P < .0001; SI, P < .0001; ACLR, P < .0001). However, the publication rate of LOE 1-2 studies only increased for RCR (P = .0003) and ACLR (P < .0001).
Conclusion:
The rate of publication of LOE 3-5 systematic reviews exponentially increased in orthopaedic sports medicine journals over the past decade, outpacing the publication rate of LOE 1-2 systematic reviews.
Keywords: systematic review, level of evidence, sports medicine, journal
Systematic reviews and meta-analyses have become increasingly important in evidence-based clinical decision making. Systematic reviews summarize the current literature using a detailed search of the literature based on a predetermined research question. Additional meta-analyses can achieve higher power and detect more nuanced differences in patient outcomes. The aggregated data can be used by clinicians to quickly see synthesized results that are more generalizable than the findings of individual studies.
However, the procedure for conducting a systematic review with or without meta-analysis is complicated and time-consuming. Thoughtful consideration is necessary regarding study question, data selection, and quantitative analysis. Methodologic reporting scores have been developed and have revealed a high error rate in the systematic review literature. 4,6,7 Dijkman et al 4 reported that although there had been an exponential increase in orthopaedic surgery systematic reviews from 1999 to 2008, there had not been an increase in study quality, and in 2008, 68% of systematic reviews had a methodologic flaw.
Furthermore, the quality of systematic reviews is dependent on the quality of the included studies. The popularization of systematic reviews has been met with controversy because of concerns that the primary literature for certain topics may not be suited for systematic review and meta-analysis. 5 Conducting a meta-analysis does not overcome the methodologic shortcomings of the studies it includes. 8 Authors must be careful not to include flawed studies because they will taint the study conclusions when the data are synthesized. Similarly, heterogeneous data may make pooled analysis invalid. As a result, the primary literature on a topic must have several high-quality studies in order to produce a high-quality meta-analysis.
The increasing focus on evidence-based medicine (EBM) has led to a larger role of quality assessment in clinical studies. A cornerstone of EBM is the level of evidence (LOE), which provides a hierarchical system for classifying the evidence of different research designs. 1 The LOE of a study can range from 1 (strong, high-quality evidence) to 5 (effectiveness of evidence not established). Several journals assign an LOE to their published articles, and authors are often asked the LOE of their studies when they submit abstracts to conferences. 1,10 Attention to LOE has had a positive effect on the orthopaedic literature, with increases in the number and proportion of LOE 1-2 papers in orthopaedic subspecialty journals. 3
Previous studies have investigated changes in LOE in the orthopaedic surgery literature. 9,11 DiSilvestro et al 5 conducted a systematic review assessing the quality of systematic reviews published in prominent sports medicine journals. Over a 5-year period, they concluded that 53% of systematic reviews were LOE 4-5 and 32% were LOE 1-2. However, there has not been a study investigating the changes in LOE of systematic reviews in sports medicine over time. Additionally, trends in the LOE of systematic reviews of important topics in sports medicine have not been previously investigated.
The purpose of this study was to assess the rate of publication of systematic reviews according to LOE within influential orthopaedic sports medicine journals and commonly researched topics in sports medicine. We hypothesized that the rate of systematic review publication has increased over the past decade, particularly LOE 3-5 studies.
Methods
An electronic search was performed using the PubMed database of studies published from January 2010 to December 2020, based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. We identified major sports medicine journals with high impact factors (IFs): American Journal of Sports Medicine (AJSM; IF, 6.202), Arthroscopy (IF, 4.77), British Journal of Sports Medicine (BJSM; IF, 13.8), Journal of Bone and Joint Surgery–American Volume (JBJS; IF, 4.57), Journal of Shoulder and Elbow Surgery (JSES; IF, 3.019), and Sports Medicine (SM Auckland; IF, 11.13). 7 These publications were chosen as a representative selection of widely read orthopaedic sports medicine journals. The only journal included with a lower IF was JSES, which was selected because of its focus on upper extremity literature. Using the advanced search tool, the records of systematic reviews published from 2010 to 2020 were extracted with the search (“Journal Name”[Journal]) AND (systematic review).
We also extracted the records of systematic reviews published on commonly researched sports medicine topics: anterior cruciate ligament reconstruction (ACLR), rotator cuff repair (RCR), meniscal repair, and shoulder instability. These topics were chosen because of their consistent presence in orthopaedic sports medicine research topics ranking within the top 4 subjects of the 100 most cited papers in Clinical Orthopedic Sports Medicine. 8 An example of the search performed was “rotator cuff repair AND systematic review.”
Systematic reviews that were either published in one of the journals of interest or studying one of the topics of interest were included. Several studies were excluded during the screening process because they were not systematic reviews, or they were neither published in the journals of interest nor studying one of the topics of interest.
The LOE was assigned to each included study by a single reviewer (T.P.). The LOE was assessed either by the listing of the LOE in the publication or by assessing the LOE of the studies according to the Oxford Centre for Evidence-Based Medicine. Studies were grouped as LOE 1-2, LOE 3-5, and nonclinical systematic reviews. Nonclinical systematic reviews were categorized as those that did not fit into the categories of LOE assessment, such as cadaveric studies, biomechanical studies, or studies on radiographic techniques. A second reviewer (C.M.) performed an independent assessment of 100 studies to assess for typographic errors. The reviewers agreed in 98% of cases.
We assessed changes in the frequency of publication of the systematic reviews, stratified by LOE grouping, by journal and by topic. We used SAS (Version 9.4; SAS Institute) and applied test of beta coefficient = 0 from a model of count = year and ran a negative binomial regression to determine the changes in publication rate between 2010 and 2020. Significance was set at P < .05.
Results
Of an initial 3115 records retrieved, 2162 systematic reviews for which the LOE could be determined were included (Figure 1). There was substantial variation in the number of systematic reviews published by the journals from 2010 to 2020. Arthroscopy and BJSM published more than 400 systematic reviews throughout the decade, while JBJS published the fewest, with only 84 (Table 1). All journals published more LOE 3-5 systematic reviews than LOE 1-2, with Arthroscopy publishing the highest proportion of LOE 3-5 studies (83.1%) and SM Auckland published the lowest proportion (53.7%). SM Auckland published the highest proportion of LOE 1-2 studies (44.4%), while JSES had the lowest (10.3%). Nonclinical systematic reviews were relatively uncommon, with JSES publishing the highest proportion (7.1%) and JBJS publishing the fewest (0%).
Figure 1.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram of study inclusion. LOE, level of evidence.
Table 1.
Distribution of Systematic Reviews According to Journal and Topic, Stratified by LOE a
| Total Studies, n | LOE 1-2 | LOE 3-5 | Nonclinical | |
|---|---|---|---|---|
| Journal | ||||
| AJSM | 236 | 23.7 (56) | 70.3 (166) | 5.9 (14) |
| Arthroscopy | 421 | 13.1 (55) | 83.1 (350) | 3.8 (16) |
| BJSM | 403 | 37.7 (152) | 60 (242) | 2.2 (9) |
| JBJS | 84 | 41.7 (35) | 58.3 (49) | 0 |
| JSES | 156 | 10.3 (16) | 82.7 (129) | 7.1 (11) |
| SM Auckland | 354 | 44.4 (157) | 53.7 (190) | 2 (7) |
| Topic | ||||
| Meniscus | 63 | 6.3 (4) | 92.1 (58) | 1.6 (1) |
| RCR | 194 | 25.3 (49) | 70.6 (137) | 4.1 (8) |
| SI | 161 | 3.1 (5) | 93.8 (151) | 3.1 (5) |
| ACLR | 430 | 27.2 (117) | 68.4 (294) | 4.4 (19) |
a Data are presented as % (No. of studies) unless otherwise indicated. ACLR, anterior cruciate ligament reconstruction; AJSM, American Journal of Sports Medicine; BJSM, British Journal of Sports Medicine; JBJS, Journal of Bone and Joint Surgery; JSES, Journal of Shoulder and Elbow Surgery; LOE, level of evidence; RCR, rotator cuff repair; SI, shoulder instability; SM Auckland, Sports Medicine.
In terms of topics and sports medicine, ACLR was the most common topic of systematic reviews (n = 430) and meniscal repair was the least popular (n = 63). Shoulder instability had the highest proportion of LOE 3-5 studies (93.8%), and ACLR had the lowest proportion (68.4%). ACLR had the highest proportion of LOE 1-2 studies (27.2%) while shoulder instability had the lowest proportion (3.1%). ACLR also had the highest proportion of nonclinical systematic reviews (4.4%), and meniscus had the lowest proportion (1.6%) (Table 1).
During the study period, the rate of publication of LOE 1-2 systematic reviews increased significantly in AJSM (P < .0001), Arthroscopy (P = .02), BJSM (P < .0001), and SM Auckland (P < .0001); however, no significant change was seen in JBJS (P = .08) or JSES (P = .15) (Table 2). The publication of LOE 3-5 systematic reviews consistently increased significantly in many of the surveyed journals (AJSM, P < .0001; Arthroscopy, P = .01; BJSM, P < .0001; JSES, P < .0001; SM Auckland, P < .0001), with the exception of JBJS (P = .57). The frequency of publication of nonclinical systematic reviews increased in BJSM (P = .01) and SM Auckland (P = .03), but not in AJSM (P = .07), JSES (P = .32), or Arthroscopy (P = .23).
Table 2.
Rate of Increased Publication of Systematic Reviews Between 2010 and 2020 According to Journal and Topic, Stratified by LOE a
| LOE 1-2 | LOE 3-5 | Nonclinical | ||||
|---|---|---|---|---|---|---|
| Rate (95% CI) | P | Rate (95% CI) | P | Rate (95% CI) | P | |
| Journal | ||||||
| AJSM | 1.30 (1.17-1.44) | <.0001 | 1.28 (1.21-1.36) | <.0001 | 1.18 (0.99-1.41) | .07 |
| Arthroscopy | 1.13 (1.02-1.25) | .02 | 1.11 (1.02-1.20) | .01 | 1.14 (0.92-1.39) | .23 |
| BJSM | 1.26 (1.16-1.37) | <.0001 | 1.20 (1.10-1.32) | <.0001 | 1.88 (1.19-2.97) | .01 |
| JBJS | 0.91 (0.82-1.01) | .08 | 1.03 (0.93-1.14) | .57 | — b | — b |
| JSES | 1.13 (0.96-1.34) | .15 | 1.29 (1.17-1.41) | <.0001 | 1.12 (0.89-1.41) | .32 |
| SM Auckland | 1.21 (1.15-1.28) | <.0001 | 1.26 (1.16-1.38) | <.0001 | 1.42 (1.03-1.95) | .03 |
| Topic | ||||||
| Meniscus | 1.21 (0.85-1.71) | .29 | 1.29 (1.17-1.42) | <.0001 | 1.99 (0.49-8.10) | .33 |
| RCR | 1.20 (1.09-1.32) | .0003 | 1.20 (1.12-1.28) | <.0001 | 1.31 (1.00-1.71) | .05 |
| SI | 1.00 (0.76-1.32) | >.99 | 1.25 (1.18-1.32) | <.0001 | 1.16 (0.86-1.55) | .33 |
| ACLR | 1.13 (1.07-1.20) | <.0001 | 1.20 (1.15-1.25) | <.0001 | 1.15 (0.99-1.34) | .06 |
a Boldface P values indicate a statistically significant increase between 2010 and 2020. ACLR, anterior cruciate ligament reconstruction; AJSM, American Journal of Sports Medicine; BJSM, British Journal of Sports Medicine; JBJS, Journal of Bone and Joint Surgery; JSES, Journal of Shoulder and Elbow Surgery; LOE, level of evidence; RCR, rotator cuff repair; SI, shoulder instability; SM Auckland, Sports Medicine.
b Model did not converge because all values were zero.
The publication rate of LOE 3-5 systematic reviews increased for all 4 topics surveyed (P < .0001 for all). However, the rate of publication of LOE 1-2 studies only increased for RCR (P = .0003) and ACLR (P < .0001). Publications of nonclinical systematic reviews did not increase on any topic (Table 2). The trends in publication of systematic review studies are demonstrated in Figure 2.
Figure 2.
Trends over time (2010-2020) in the publication of systematic review studies based on level of evidence, by (A) journal and (B) topic. ACLR, anterior cruciate ligament reconstruction; AJSM, American Journal of Sports Medicine; BJSM, British Journal of Sports Medicine; JBJS, Journal of Bone and Joint Surgery; JSES, Journal of Shoulder and Elbow Surgery.
Discussion
This study demonstrates that, in general, publications for LOE 3-5 systematic reviews have been increasing over the past decade, with the percentage of LOE 3-5 systematic reviews outpacing LOE 1-2 systematic reviews for all high-impact sports journals assessed. All the journals included in this study had exponential increases in the publication of LOE 3-5 systematic reviews (AJSM, P < .0001; Arthroscopy, P = .01; BJSM, P < .0001; JSES, P < .0001; SM Auckland, P < .0001), except for JBJS (P = .57). Similarly, the publication of LOE 1-2 systematic reviews exponentially increased (AJSM, P < .0001; Arthroscopy, P = .02; BJSM, P < .0001; SM Auckland, P < .0001) except for JBJS (P = .08) and JSES (P = .15), but still comprised a lower percentage compared with LOE 3-5 reviews. The sports medicine topics included in this study saw exponential increases in LOE 3-5 systematic reviews (meniscal repair, P < .0001; RCR, P < .0001; shoulder instability, P < .0001; ACLR, P < .0001). However, only RCR (P = .0003) and ACLR (P < .0001) had increases in LOE 1-2 systematic reviews. The publication of nonclinical systematic reviews increased in BJSM (P = .01) and SM Auckland (P = .03), and the only topic that had an increase in nonclinical systematic reviews was RCR (P = .05). Overall, our study demonstrates the overall exponential increase in systematic reviews published in leading sports medicine journals and in common sports medicine topics.
A study comparing systematic reviews published in AJSM, JBJS, Arthroscopy, Sports Health, and Knee Surgery, Sports Traumatology, Arthroscopy demonstrated that journals with a higher IF published systematic reviews with higher PRISMA and Assessment of Multiple Systematic Reviews (AMSTAR) scores. 5 They also reported that between 2009 and 2013, 53% of studies included LOE 4-5 studies and 32% were LOE 1-2. Comparatively, 28.5% of systematic reviews in our study were LOE 1-2, while 68.1% were LOE 3-5 from 2010 to 2020. Our findings suggest that there has continued to be an increasing divide between the publication rate of LOE 1-2 and LOE 3-5 systematic reviews.
Of the included journals, only JBJS did not publish more LOE 1-2 or LOE 3-5 systematic reviews in the past decade. As of May 31, 2021, JBJS has added additional criteria for the consideration of this matter for publication in their journals. These criteria include meta-analyses that only include randomized controlled trials that are sufficiently homogeneous in terms of inclusion and exclusion criteria, attempting to obtain unpublished data related to their investigation, adhering to the PRISMA guideline. 13
The publication of systematic reviews on meniscal repair and shoulder instability has demonstrated exponential increases in LOE 3-5 studies, but no increase in LOE 1-2 studies. This is possibly because of a lack of level 1 or 2 studies in the primary literature. Investigators and readers of orthopaedic sports medicine literature should be encouraged to produce high-quality studies on these research topics that can later be used for high-powered meta-analysis.
A difficulty of conducting LOE 1-2 studies and systematic reviews has been a limitation in the orthopaedic sports medicine literature. 2 Fortunately, well-designed LOE 3-5 systematic reviews can still have clinical utility. 11 Identifying high-quality LOE 3-5 systematic reviews necessitates defining strict criteria for systematic reviews to make it more transparent to readers and provide them with the opportunity to judge if the conclusion of the study is reliable and could be used in practice. The AMSTAR methodologic criteria and PRISMA guidelines should be used to evaluate methodologic and reporting quality, respectively. Publication bias presents an additional challenge of interpreting systematic reviews. Scott et al 12 demonstrated that publication bias was only assessed in 19.5% of systematic reviews in the top 10 orthopaedic journals. Publication bias can be assessed by reporting the heterogeneity of included studies using forest plots, the Egger regression, and the Duval and Tweedie trim and fill method. 12 These quality measures do not guarantee the validity of a study’s conclusions, but help readers evaluate the strengths and weaknesses of a study when they are considering applying recommendations to their practice.
Strengths and Limitations
There are some strengths to this study. This study surveyed systematic reviews in influential sports medicine journals and topics and assessed their LOE. One of the most common future directions of systematic reviews is the need for more high-quality LOE 1-2 studies. This study is to our knowledge the first to analyze the most popular topics in sports medicine and describe their trends of systematic review publication. This study also has limitations. Several systematic reviews did not state their LOE in the paper and did not state the study design of the included papers. This deficiency led to several systematic reviews being excluded. Health practitioners should be aware of restrictions in systematic reviews and use them in practice, especially with regard to LOE 3-5 systematic reviews that have a contradictory conclusion compared with individual randomized clinical trials. We believe that journals should adopt a requirement of PRISMA guideline reporting, and researchers may need to consider only LOE 1-2 for systematic review and meta-analysis, when there are sufficient data to do so.
Conclusion
The publication rate of LOE 3-5 systematic reviews has exponentially increased in orthopaedic sports medicine journals over the past decade and has outpaced the publication rate of LOE 1-2 systematic reviews. Common topics in sports medicine have consistently seen increases in publication rate of LOE 3-5 systematic reviews, but not LOE 1-2 studies. LOE 1-2 primary research studies are necessary to produce homogeneous cohorts and outcomes necessary for high-quality systematic reviews on important sports medicine topics. Furthermore, authors should follow methodologic criteria such as AMSTAR and PRISMA to allow readers to assess systematic review quality and implement evidence-based recommendations into practice.
Acknowledgment
The authors thank Ting Cong, MD, and Justin Greiner, MD, for their assistance with manuscript review.
Footnotes
Final revision submitted March 5, 2022; accepted June 6, 2022.
One or more of the authors has declared the following potential conflict of interest or source of funding: A.L. has received consulting fees from Arthrex and Tornier/Stryker. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
References
- 1. Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128(1):305–310. doi:10.1097/PRS.0b013e318219c171 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Chaudhry H, Mundi R, Singh I, Einhorn TA, Bhandari M. How good is the orthopaedic literature? Indian J Orthop. 2008;42(2):144–149. doi:10.4103/0019-5413.40250 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Cunningham BP, Harmsen S, Kweon C, et al. Have levels of evidence improved the quality of orthopaedic research? Clin Orthop Relat Res. 2013;471(11):3679–3686. doi:10.1007/s11999-013-3159-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Dijkman BG, Abouali JA, Kooistra BW, et al. Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity? J Bone Joint Surg Am. 2010;92(1):48–57. doi:10.2106/JBJS.I.00251 [DOI] [PubMed] [Google Scholar]
- 5. DiSilvestro KJ, Tjoumakaris FP, Maltenfort MG, Spindler KP, Freedman KB. Systematic reviews in sports medicine. Am J Sports Med. 2016;44(2):533–538. doi:10.1177/0363546515580290 [DOI] [PubMed] [Google Scholar]
- 6. Dosenovic S, Jelicic Kadic A, Vucic K, Markovina N, Pieper D, Puljak L. Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR. BMC Med Res Methodol. 2018;18(1):37. doi:10.1186/s12874-018-0493-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Gagnier JJ, Kellam PJ. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am. 2013;95(11):e771–e777. doi:10.2106/JBJS.L.00597 [DOI] [PubMed] [Google Scholar]
- 8. Garg AX, Hackam D, Tonelli M. Systematic review and meta-analysis: when one study is just not enough. Clin J Am Soc Nephrol. 2008;3(1):253–260. doi:10.2215/CJN.01430307 [DOI] [PubMed] [Google Scholar]
- 9. Judy RP, Shin JJ, McCrum C, Ayeni OR, Samuelsson K, Musahl V. Level of evidence and authorship trends of clinical studies in knee surgery, sports traumatology, arthroscopy, 1995-2015. Knee Surg Sports Traumatol Arthrosc. 2018;26(1):9–14. doi:10.1007/s00167-017-4801-6 [DOI] [PubMed] [Google Scholar]
- 10. Lerman DM, Cable MG, Thornley P, et al. Has the level of evidence of podium presentations at the Musculoskeletal Tumor Society annual meeting changed over time? Clin Orthop Relat Res. 2017;475(3):853–860. doi:10.1007/s11999-016-4763-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Luksameearunothai K, Chaudhry Y, Thamyongkit S, Jia X, Hasenboehler EA. Assessing the level of evidence in the orthopaedic literature, 2013-2018: a review of 3449 articles in leading orthopaedic journals. Patient Saf Surg. 2020;14:23. doi:10.1186/s13037-020-00246-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Scott J, Checketts JX, Cooper CM, Boose M, Wayant C, Vassar M. An evaluation of publication bias in high-impact orthopaedic literature. JBJS Open Access. 2019;4(2):e0055. doi:10.2106/JBJS.OA.18.00055 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Swiontkowski M. Meta-analyses and systematic reviews: JBJS policy revisited. J Bone Joint Surg Am. 2021;103(10):849. doi:10.2106/JBJS.21.00233 [DOI] [PubMed] [Google Scholar]