Discordant Systematic Reviews: Which to Believe?

Introduction

Systematic reviews of trials are considered to represent the highest level of evidence.¹ As such they have become prevalent in all disciplines, with numbers increasing each year. The spine literature mimics that trend (see Figure 1). As more and more systematic reviews are being published, clinicians encounter important discordant results from different systematic reviews on the same topic. When this happens, which systematic review should be believed? In this article, we identify a few important issues to consider when deciding which discordant review is most appropriate for clinical decision making.

Figure 1.

The number of systematic review citations by year utilizing the following search strategy in PubMed on November 132 019 (Spine OR Spinal OR *Lumbar OR Radiculopathy OR Myelopathy OR Kyphoplasty OR Vertebroplasty OR Lumbosacral) AND Systematic Review.

Issues to Consider

Not Every Systematic Review Is a Systematic Review

It has become challenging for consumers of the literature (and authors, reviewers, and editors!) to agree on a common or shared understanding of the various types of reviews, their nomenclature, and their value.² Often we encounter articles that are characterized as “systematic review” but do not have the essential characteristics of a systematic review as defined in publication guidelines.^3-6 Not infrequently we encounter those who erroneously believe that a systematic search equals a systematic review. While an exhaustive search to identify all possible publications on a specific topic is important, it is only one aspect of a much larger process. Note how Cochrane describes the characteristics of a systematic review³:

A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made. The key characteristics of a systematic review are:

• a clearly stated set of objectives with pre-defined eligibility criteria for studies;

• an explicit, reproducible methodology;

• a systematic search that attempts to identify all studies that would meet the eligibility criteria;

• an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias; and

• a systematic presentation, and synthesis, of the characteristics and findings of the included studies.”

Make sure that the discordant systematic reviews meet the criteria of a systematic review. One or more may not be a systematic review at all.

Not All Discordant Systematic Reviews Address the Same Clinical Question

Whether discordant systematic reviews address the same clinical question can best be determined by comparing the clinical questions in terms of the populations, interventions, comparators, and outcomes examined in each review. Often there are subtle differences between the reviews published, and compared with the information sought by the clinical decision maker. Close examination is needed to determine these differences. When this happens, select the review containing the clinical question that most closely addresses the problem of interest.

Not All Discordant Systematic Reviews Are of the Same Quality

Systematic reviews are classified as observational research studies. The quality of the study, like any research study, varies depending on how rigorously the research team attempts to limit bias. In evaluating the quality of discordant systematic reviews, one can use the 16-item AMSTAR 2 tool (checklist).⁶ In doing so, pay particular attention to 7 critical areas listed in Table 1. From these critical questions, a level of confidence in the results of the reviews can be obtained (Table 2).

Table 1.

Seven Critical Domains for Assessing Systematic Review Quality Using the AMSTAR 2⁶.

AMSTAR 2 Item Number	Critical Domains
Item 2	The research questions and the review study methods should have been planned ahead of conducting the review.
Item 4	At least 2 bibliographic databases should be searched and supplemented by checking published reviews and references of studies found. Key words should be reported.
Item 7	A complete list of potentially relevant studies should be provided with justification for the exclusion of each.
Item 9	Review authors should have used a systematic approach to RoB assessment, preferably with a properly developed rating instrument.
Item 11	If meta-analysis was justified, the authors should include a statement of the principles used on deciding whether to perform a meta-analysis, an explanation of the decision to use a fixed or random effects model, and how heterogeneity was investigated.
Item 13	Authors should include a discussion of the impact of RoB in the interpretation of the results of the review.
Item 15	If quantitative synthesis was performed, the authors should carry out an adequate investigation of publication bias and discuss its likely impact on the results of the review.

Table 2.

Rating Overall Confidence in the Results of a Systematic Review From AMSTAR ⁶.

Confidence in Results	Explanation	Criteria
High	Provides an accurate and comprehensive summary of the results of the available studies that address the question of interest.	No or one noncritical weakness
Moderate	The systematic review has more than one weakness but no critical flaws. It may provide an accurate summary of the results of the available studies that were included in the review.	More than one noncritical weaknessa
Low	The review has a critical flaw and may not provide an accurate and comprehensive summary of the available studies that address the question of interest.	One critical flaw with or without noncritical weaknesses
Critically low	The review has more than one critical flaw and should not be relied on to provide an accurate and comprehensive summary of the available studies.	More than one critical flaw with or without noncritical weaknesses

^a Multiple noncritical weaknesses may diminish confidence in the review and it may be appropriate to move the overall appraisal down from moderate to low confidence.

Summary

• More and more spine-related systematic reviews are being published, increasing the likelihood that clinicians may encounter discordant results from different reviews on the same topic. When this happens, the clinical decision maker should consider important issues when deciding which review is most appropriate. One may be able to exclude several from consideration, making the process much more efficient.

• Though labeled as systematic reviews, some reviews do not have the necessary characteristics of a systematic review as defined in publication guidelines. A systematic review contains key characteristics to include predefined eligibility criteria, reproducible methodology, identification of all studies meeting criteria, assessment of risk of bias, and systematic presentation and synthesis of results.

• Discordant reviews may not address the same clinical questions. This can be determined by carefully comparing the key questions in terms of populations, interventions, comparators, and outcomes examined in each review.

• Systematic reviews are observational research studies. They can be assessed for quality using a quality checklist such as the AMSTAR 2.