Declining Quality of Systematic Reviews in Orthopaedic Sports Medicine: An Updated Systematic Review

Abstract

Purpose

To evaluate the quality and characteristics of systematic reviews, including meta-analyses, in the clinical orthopaedic sports medicine literature from 2015 to 2019 and to compare the results to previous findings from a similar analysis from 2009 to 2013.

Methods

All clinical orthopaedic sports medicine and meta-analyses published from 2015 to 2019 published in The American Journal of Sports Medicine, Arthroscopy, The Journal of Bone and Joint Surgery, Knee Surgery, Sports Traumatology, Arthroscopy, and Sports Health were reviewed. These were evaluated according to guidelines from Oxford Centre for Evidence-Based Medicine, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement, and the Assessment of Multiple Systematic Reviews tool. Results were compared to the quality of publications from 2009 to 2013.

Results

A total of 516 systematic reviews and meta-analyses were included in this study. Of these, 17.1% of studies included Level I or II evidence only, whereas 79.1% included Level IV or V studies. When compared to the previous study from 2009 to 2013, which demonstrated 32% of Level I or II evidence studies and 53% Level IV or V, there was a significant decrease in the level of evidence in the more recent study period (P < .001). The average Preferred Reporting Items for Systematic Reviews and Meta-Analyses scores were 81% and the average Assessment of Multiple Systematic Reviews scores 56%, which are declines from 87% and 73%, respectively (P < .001, P < .001).

Conclusions

There has been a significant increase in the volume of systematic reviews and meta-analyses published in orthopaedic sports medicine. This has coincided with significant declines in the level of evidence, as well as declines in methodologic and reporting quality.

Clinical Relevance

Clinicians increasingly use systematic reviews to drive their treatment decisions. Therefore, the quality of systematic reviews in orthopaedic sports medicine merits assessment.

Systematic reviews are powerful, evidence-based tools that can provide a basis for rational decision-making in medical practice.¹ A systematic review is defined as a scientific investigation that compiles, critically appraises, and synthesizes the results of primary investigations addressing a specific topic or research question. A quantitative systematic review, also known as a meta-analysis, uses statistical methods to combine the results of multiple studies for analysis.² These studies, although a vital resource, can be flawed. The reporting quality can vary, thus limiting the ability to accurately assess the information presented.³

Quality assessment analyses of systematic reviews have been completed previously across multiple medical disciplines. Literature within the fields of pediatrics, emergency medicine, and orthopaedic surgery have all demonstrated unsatisfactory quality of reporting in their systematic reviews.4, 5, 6 A 2020 evaluation of 96 systematic reviews in spine surgery reported a subpar mean score per the Assessment of Multiple Systematic Reviews (AMSTAR) guidelines.⁷ The most recent quality assessment of systematic reviews related to sports medicine was published in 2015 and evaluated literature from 2009 to 2013. It found studies relied on primary literature with an evidence level of IV or lower nearly 53% of the time; however, the systematic reviews had high AMSTAR (73%) and Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) scores (87%).⁸

A recent assessment suggests that more than 8,000 systematic reviews are indexed in MEDLINE annually, indicating a need for more regular evaluations.⁹ In sports medicine alone, queries in MEDLINE reveal that the number of systematic reviews related to sports medicine have exploded from fewer than 10 per year in 2012 to 60 in 2017.¹⁰ Garner et al.¹¹ suggested that updated systematic reviews are appropriate when the review addresses a current question, uses valid methods, is well conducted, and if there is new relevant information on the topic. Given this increase in systematic reviews, including in field of sports medicine, this article addresses a valid question, incorporates new data, and can potentially change our previous findings, thus fitting the criteria for an update of the previous review. The purposes of this review are to evaluate the quality and characteristics of systematic reviews, including meta-analyses, in the clinical orthopaedic sports medicine literature from 2015 to 2019 and to compare the results to previous findings from a similar analysis from 2009 to 2013. We hypothesized that there would be no change in the quality of systematic reviews conducted from 2009 to 2013 to 2015 to 2019.

Methods

We sought to identify all systematic reviews and meta-analysis published in a 5-year period (2015-2019) from 5 orthopaedic journals in the subspecialty of orthopaedic sports medicine. To ensure comparability of results with the previous publication, we followed the same established protocols wherever possible.⁸

Eligibility Criteria

Inclusion criteria for this review includes a paper be: (1) either a meta-analysis or systematic review; (2) published between 2015 and 2019; (3) published in The American Journal of Sports Medicine (AJSM), Arthroscopy, Journal of Bone and Joint Surgery (JBJS), Knee Surgery, Sports Traumatology, Arthroscopy (KSSTA), or Sports Health; and (4) have a clinical sports medicine orthopaedic topic. The 5 journals were selected to maintain consistency with the previous publication. Previously, they were selected for their high impact factor in the field of orthopaedic sports medicine per the ISI Thomas Reuters Journal Citation Reports. Exclusion criteria included animal, cadaveric, and basic science articles; editorials, surveys, special topics, letters to the editor, correspondence; non–sports-related topics such as total joint replacement, infection after open long bone fractures, osteonecrosis, and the effect of smoking.

Paper Identification

Two authors independently performed the identification process. All 5 journals were manually scanned via their online databases to identify articles that met the inclusion criteria. All journals were downloaded electronically and reviewed. The independent lists were compared for disparities. Articles were then examined for exclusion based on established criteria. For disagreements at any stage, the senior author rendered a final decision as to inclusion or exclusion.

Assessment of Reporting Quality

The PRISMA statement was used to evaluate the reporting quality of reviewed papers. Developed in 2009, a 27-item checklist aims to improve the reporting of meta-analyses.³ This checklist is available as item 1 of Appendix 1, available at www.arthroscopyjournal.org. All items on the checklist are answered with “yes,” “no,” “don’t know,” and “N/A.” “Yes” meant the item had been fulfilled; “no” meant the item was unfulfilled; “don’t know” meant that due to unclear language or incomplete reporting, it is inconclusive as to whether the item had been fulfilled; and “N/A” meant that the item was not applicable to the paper being evaluated. Scoring is performed by dividing the number of “yes” answers by the total number of questions, including those that were answered “N/A,” thus representing a percentage of possible score earned. In addition, the lowest level of evidence included in each systematic review was recorded.

Assessment of Methodologic Quality

The AMSTAR tool was used to evaluate the methodologic quality of reviewed papers. Originally published in 2007, the tool received an update as the AMSTAR2 in 2017.¹² The 2007 version was used for this review to maintain consistency with the previous publication. The AMSTAR is an 11-item checklist, intended to assess the methodologic quality of systemic reviews in a rapid and reproducible manner.¹³ This checklist is available as item 2 of Appendix 1. All items on the checklist are answered with “yes,” “no,” “can’t answer,” and “N/A.” These responses have the same meaning to their PRISMA counterparts. Scoring is performed in a similar manner, also representing a percentage of possible score earned.

Data Extraction

All data were extracted into an Excel (Microsoft Corp., Redmond, WA) spreadsheet. Each paper was evaluated with the PRISMA and AMSTAR checklists, with score recorded. The lowest level of evidence, as defined by the Oxford Centre for Evidence-Based Medicine, was noted.¹⁴ This was recorded as “high” if the review included Level I or II studies only; it was recorded as “low” if it included Level IV or V studies, as reported in the previous publication. In addition, the date of publication, journal of publication, topic for each paper, and total number of research articles published were also noted. Topics were defined as knee, shoulder, foot, hip, wrist, elbow, or multijoint/other.

Data Analysis

In a design identical to the previous publication, PRISMA and AMSTAR scores were calculated as the total number of “yes” responses divided by the total number of “yes,” “no,” and “don’t know” responses. This differs from the standard scoring method, dividing the number of “yes” answers by the total number of questions, including those that were answered “N/A.” Under this method, a study that is only a systematic review would be penalized for not including items such as the method used for the synthesis of results or a χ² value. Since these questions are only relevant to a meta-analysis that combines data, our previous publication argued that a systematic review should not be judged by these criteria. For this reason, questions that are only relevant to a meta-analysis were deleted from the analysis for systematic reviews. Average scores were tabulated by year and by journal. The proportion of systemic reviews and meta-analysis to the total number of published articles was noted. These were compared with the results of the previous publication. Statistical analysis was performed with SPSS (version 27.0; IBM Corp., Armonk, NY), using t-test, analysis of variance, and χ² analysis where appropriate. A P-value of <.05 was considered significant.

Results

A total of 628 systemic reviews and meta-analyses were identified from 2015 to 2019 via initial journal review. In total, 112 articles were excluded based on established criteria, leaving 516 articles for final inclusion. These systematic reviews and meta-analysis represent 6.9% of all articles published in these journals during time period. This compares to 200 articles identified from 2009 to 2013 in the first review, which were 3.2% of all published articles. This represented a statistically significant increase in the proportion of sports medicine systemic reviews and meta-analyses (P < .001). In total, 109 sports medicine reviews were published in 2015 (7.4% of all articles published in 2015), 116 in 2016 (7.6%), 85 in 2017 (5.6%), 97 in 2018 (6.7%), and 109 in 2019 (7.5%). There was no significance difference in the proportion of reviews published by year.

By subject, 253 knee (49.0%), 106 shoulder (21.5%), 64 multijoint/other (12.4%), 54 hip (10.5%), 24 foot/ankle (4.7%), 11 elbow (2.1%), and 1 wrist (0.2%) reviews were published from 2015-2019 (Table 1). This represents a statistically significant (P < .001) shift in published sports medicine systemic review and meta-analysis subject matter from 2009 to 2013. The biggest proportional shifts occurred in multijoint/other (12.4% up from 3.0%), knee (49.0% down from 58.5%), and hip (10.5% up from 5.5%).

Table 1.

Total Manuscripts by Subject

Subject	2009-2013	2015-2019
Total	200	516
Knee	117 (58.5%)	253 (49.0%)
Shoulder	48 (24.0%)	106 (21.5%)
Foot/ankle	15 (7.5%)	24 (4.7%)
Hip	11 (5.5%)	54 (10.5%)
Wrist	2 (1.0%)	1 (0.2%)
Elbow	1 (0.5%)	11 (2.1%)
Multijoint/other	6 (3.0%)	64 (12.4%)

Level of evidence of published reviews did not change throughout the study period from 2015-2019 (Table 2). In total, 88 high-level (17.1%) and 408 low-level evidence (79.1%) papers were published in the study period, compared with 63 high-level (31.5%) and 108 low-level evidence (54.0%) in 2009 to 2013. There was a statistically significant decline in the level of evidence (P < .001) of sports medicine systemic review and meta-analyses between study periods. The level of evidence declined at the individual journal level for AJSM (P = .012), Arthroscopy (P < .001), KSSTA (P < .001), and Sports Health, whereas there was no change for JBJS from 2009-2013 to 2015-2019.

Table 2.

Manuscripts With High and Low Levels of Evidence, Average PRISMA Scores, and Average AMSTAR Scores (in Percentages) by Year

	2015	2016	2017	2018	2019	Total '09-'13	Total '15-'19
Total number of manuscripts	109	116	85	97	109	200	516
High level of evidence∗	20 (18%)	18 (17%)	13 (15%)	20 (21%)	17 (16%)	63 (31.5%)	88 (17.1%)
Low level of evidence∗	84 (77%)	95 (82%)	68 (80%)	71 (73%)	90 (83%)	108 (54.0%)	408 (79.1%)
PRISMA score, average	76	79	81	84	82	87	80
AMSTAR score, average	53	59	58	63	61	73	59

AMSTAR, Assessment of Multiple Systematic Reviews; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

^∗Data presented as: number of manuscripts (percentage of manuscripts).

By journal (Table 3), 143 reviews were published in AJSM from 2015 to 2019 (7.7% of all articles published in AJSM), 235 in Arthroscopy (19.2%), 13 in JBJS (0.8%), 87 in KSSTA (3.6%), and 38 in Sports Health (11.8%) from 2015 to 2019. For contrast, the reviews from 2009 to 2013 are summarized in Table 2. Four of the journals, AJSM, Arthroscopy, KSSTA, and Sports Health, had a significant increase in the proportion of published reviews (P < .001, P < .001, P = .036, P = .010, respectively), whereas JBJS (P = .200) did not.

Table 3.

Manuscripts With High and Low Levels of Evidence, Average PRISMA Scores, and Average AMSTAR Scores (in Percentages) by Journal

	AJSM	Arthroscopy	JBJS-A	KSSTA	Sports Health
Number of manuscripts (2009-2013)	34	90	21	36	19
High level of evidence∗	10 (30%)	22 (24%)	12 (58%)	6 (17%)	13 (68%)
Low level of evidence∗	21 (62%)	57 (63%)	5 (24%)	20 (56%)	5 (27%)
PRISMA score, average	87	89	89	84	81
AMSTAR score, average	75	74	82	68	69
Number of Manuscripts (2015-2019)	143	235	13	87	38
High level of evidence∗	27 (19%)	33 (14%)	4 (31%)	12 (14%)	12 (32%)
Low level of evidence∗	109 (76%)	193 (82%)	8 (62%)	73 (84%)	25 (66%)
PRISMA score, average	80	81	84	79	79
AMSTAR score, average	58	60	57	56	62

AJSM, The American Journal of Sports Medicine; AMSTAR, Assessment of Multiple Systematic Reviews; JBJS-A, Journal of Bone and Joint Surgery, American Volume; KSSTA, Knee Surgery, Sports Traumatology, Arthroscopy; PRISMA, Preferred Reporting Items for Systematic Reviews.

^∗Data presented as: Number of manuscripts (percentage of manuscripts).

The average PRISMA score for all reviews from 2015 to 2019 was 81, compared with 87 in 2009-2013 (P < .001). Scores in the current study period did not differ by journal (P = .903). They did increase by year (P < .001), starting at 76 in 2015, 79 in 2016, 81 in 2017, 84 in 2018, and 84 in 2019. 21 of the 27 individual checklist items were changed by a significant degree from the previous study. Of those, items 5, 10, 12, 13, 16, 19, 23, and 26 changed by more than 10% in completion rate. Items 15 (risk of bias across studies) and 22 (results of risk of bias across studies) remained the 2 lowest scored items (16.9% and 17.3% respectively), similar to 2009-2013 (23.0% and 17.4%).

Average AMSTAR scores also significantly declined, from 73 in 2009 to 2013 to 57 in 2015 to 2019 (P < .001). Scores in the current study period did not differ by journal (P = .364) but did differ by year (P = .004). There was a general trend upwards from 51 in 2015, 57 in 2016, 56 in 2017, 62 in 2018, 59 in 2019. Every checklist item significantly differed from the previous study. Of those, items 1, 4, 7, 8, and 9 differed by more than 10% completion rate. Items 5 (a list of included/excluded studies) and 10 (assess publication bias) were the 2 lowest scored items in the previous study, at 28.0% and 19.9%, respectively, and remained low at 23.6% and 29.5%. Item 4 (status of publication used in inclusion) dropped into the second lowest completed item, from 99.5% to 23.6%.

Discussion

The results of our study found that while the number of systematic reviews and meta-analyses published increased from 2009 to 2013 to 2015 to 2019, their level of evidence and quality declined. These systemic reviews and meta-analyses are used by orthopaedic sports medicine physicians, like many other specialties, to guide decision making and patient treatment. A thorough review harnesses evidence-based data gathering tools and can synthesize data from multiple studies to draw conclusions on precise questions, such as specific diagnoses or techniques. While useful tools, they are only as powerful as the quality of the included primary literature and methodological rigor of the articles included.^3,13

Previous analysis indicated rising trends in the level of evidence for studies published in orthopaedics throughout the first decade of the 21st century.¹⁵ A subsequent publication found that this trend may be relapsing.¹⁶ This reflects our findings for systematic reviews and meta-analyses in sports medicine. In total, 31.5% of reviews from 2009 to 2013 were of high-level evidence, but that number plummeted to 17.1% of reviews from 2015 to 2019. This is occurring in the context of nearly double (3.2%-6.9%) the proportion of articles published in the journals studied. The quality of evidence is important to note when drawing conclusions that impact clinical decision making. Lower-level studies are, by definition, at greater risk of bias and improper conclusions. This decreasing trend in evidence may mean that systematic reviews and meta-analyses are increasingly less impactful or relevant to physicians. It is difficult to know if this trend reflects changes to evidence in primary literature, author decisions, or both. Luksameearunothai et al.¹⁶ found that the level of evidence for published articles in 3 orthopaedic journals decreased from 2013 to 2018. This finding mirrors the decrease in level of evidence found by our study. Some of this decrease may be attributable to adaptation of changes to the level of evidence scale, including by JBJS in January 2015.¹⁷ Decreased level of evidence in primary literature will ultimately lead to decreased levels in the reviews based off them. This decreased level of evidence in primary literature may also be due to the explosion in sports medicine reviews articles,¹⁰ with authors exploring increasingly specific questions, which have lower evidence levels of literature available. With increasing numbers of publications from which to review, authors may be exploring increasingly niche topics in their systematic reviews. These will likely have fewer high-quality studies to draw from and thus the resulting review will have a lower level of evidence.

The average PRISMA score decreased from 87 to 81 between study periods. Paradoxically, both studies reported an increasing trend in PRISMA scores throughout the duration of their respective designated study periods. There is no study on interobserver agreement for PRISMA. The average AMSTAR score declined from 73 to 59 between study periods. Similar to the PRISMA, concerns about inter-relater reliability exist, and Pieper et al.¹⁸ demonstrated there can be wide variations in inter-relater reliability depending on the pair of reviewers involved. Regardless, as found in the previous study, the quality of reporting and methodology for orthopaedic sports medicine remains greater, if not in line, with orthopaedics in general. Zhi et al.¹⁹ reported an average PRISMA score of 77 of leading orthopaedic journals, Belloti et al.²⁰ found an average PRISMA and AMSTAR scores of 59 in systematic reviews on distal radius fractures, and Sathish and Eswar⁷ published a AMSTAR score of 68 of reviews in spine surgery.

Authors, readers, and publishers should consider level of evidence, quality of reporting, and methodologic quality measures of systematic reviews and meta-analyses. It is difficult to say which is more important for determining the quality of reviews. Thus, publishers should consider requiring level of evidence, PRISMA, and AMSTAR scores within the article abstract so clinicians can be properly alerted to how that review should guide decision-making. Systemic pressures such as this may push authors to improve the methodologic and reporting quality of their work. Authors should attempt to perform reviews with high level of evidence whenever possible, although this may not be feasible for niche topics. Authors should also consider methodological and reporting rigor when posing clinical questions and designing reviews. Properly established protocols may be especially important with ongoing authorship trends. The mean number of authors per article and proportion of non-MD, including more bachelor’s degree first authors, has been increasing in orthopaedic sports medicine literature.²¹ While increased interest in orthopaedic sports medicine research is welcomed, proper guidance and mentorship will be important to ensure the continued quality of publications.

Limitations

This study is not without limitations. This review was limited to 5 relatively high-quality journals. Therefore, the study may not be generalizable to the level of evidence, methodologic, and reporting quality of orthopaedic sports medicine literature as a whole. In addition, this study used the outdated AMSTAR checklist to maintain consistency with the previous study, despite the AMSTAR2 being available to address concerns about the original checklist. We reported scores based on a modified scoring criterion to maintain consistency with the prior study. While we believe this reflects a more accurate score for systematic reviews, it limits the ability to draw comparisons to other similar reviews. As previously stated, there are no data on interobserver agreement for PRISMA, and studies on AMSTAR report that there can be wide variations in inter-relater reliability.¹⁸ This may limit the comparisons that are able to be drawn between the 2 study periods. Finally, this was a review of articles relevant to orthopaedic sports medicine. Studies that were excluded based on topic may have a significant impact on our evaluated measures of quality or volume/proportion of published reviews.

Conclusions

There has been a significant increase in the volume of systematic reviews and meta-analyses published in orthopaedic sports medicine. This has coincided with significant declines in the level of evidence, as well as declines in methodologic and reporting quality.

Appendix.

Item 1. PRISMA Checklist

Title

• (1)Title

Abstract

• (2)Structured Summary

Introduction

• (3)Rationale
• (4)Objectives

Methods

• (5)Protocol and registration
• (6)Eligibility criteria
• (7)Information sources
• (8)Search
• (9)Study selection
• (10)Data Collection Process
• (11)Data items
• (12)Risk of bias in individual studies
• (13)Summary measures
• (14)Synthesis of results
• (15)Risk of bias across studies
• (16)Additional analyses

Results

• (17)Study selection
• (18)Study characteristics
• (19)Risk of bias within studies
• (20)Results of individual studies
• (21)Synthesis of results
• (22)Risk of bias across studies
• (23)Additional analysis

Discussion

• (24)Summary of evidence
• (25)Limitations
• (26)Conclusions

Funding

• (27)Funding

  Adapted from: Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339.

Item 2. AMSTAR1 Checklist

• (1)Was an 'a priori' design provided?
• (2)Was there duplicate study selection and data extraction?
• (3)Was a comprehensive literature search performed?
• (4)Was the status of publication (i.e. grey literature) used as an inclusion criterion?
• (5)Was a list of studies (included and excluded) provided?
• (6)Were the characteristics of the included studies provided?
• (7)Was the scientific quality of the included studies assessed and documented?
• (8)Was the scientific quality of the included studies used appropriately in formulating conclusions?
• (9)Were the methods used to combine the findings of studies appropriate?
• (10)Was the likelihood of publication bias assessed?
• (11)Was the conflict of interest included?

  Adapted from: Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10.