Criteria which may decrease confidence in results |
|
|
|
|
Limitations to study quality (risk of bias) |
Reflects the (lack of) internal validity of the study. The foremost quality criterion in science |
Agree with the criterion |
Decrease with 1 or 2 levels if serious limitations (–1) or very serious limitations (–2) to study quality |
Decreasing with 1 or 2 levels is appropriate. In some cases decreasing with even 3 levels (e.g. evidence from high to very low) is justified |
Inconsistency |
Reflects the (lack of) consistency of the results of a study |
Agree with the criterion |
Decrease with 1 or 2 levels if inconsistency is serious (–1) or very serious (–2) |
Decreasing with 1 or 2 levels is appropriate |
Indirectness of evidence |
All studies synthesized in a systematic review should have similar patient populations, interventions, control interventions, and outcomes. It is not appropriate to do a meta-analysis combining direct and indirect evidence posing different hypotheses, except in network meta-analyses |
Mostly disagree with the criterion |
Decrease with 1 or 2 levels if serious (–1) or very serious indirectness (–2) |
Decreasing level of evidence is not appropriate when based on summarizing results from incommensurable studies. Network meta-analyses may allow decisions for decreasing level of evidence based on indirectness |
Imprecision |
Reflects random error in outcome estimates. The wideness of confidence intervals is one result of a study or meta-analysis and should not be used as a quality criterion |
Disagree with the criterion |
Decrease with 1 or 2 levels if serious imprecision (–1) or very serious imprecision (–2) |
Decreasing level of evidence based on degree of random error in the outcome estimates is not appropriate; the limitation shown by wide confidence intervals is a result of a systematic review |
Probability of publication bias |
Selective reporting of outcomes is a matter of internal validity of the study and belongs to the ‘limitations to study quality’ criterion. When individual studies are not at all published, the results of a systematic review are potentially biased |
Agree with the criterion |
Decrease with 1 or 2 levels if publication bias likely (–1) or very likely (–2) |
Decreasing level of evidence is appropriate |
Criteria which may increase confidence in results |
|
|
|
|
Large magnitude of effect |
This is a result of a study or meta-analysis and should not be used as a quality criterion. Large magnitude of effect may imply a high risk of biased results rather than increased confidence in results |
Disagree with the criterion |
Increase with 1 or 2 levels if large (+ 1) or very large (+ 2) evidence of association |
Increasing level of evidence based on large magnitude of effect is not appropriate, because of a risk for biased conclusions |
Dose-response gradient |
Dose-response gradient often exists in studies assessing etiology of disease, but effectiveness of an intervention usually does not show a linear dose-response pattern |
Mostly disagree with the criterion |
Increase with 1 level if evidence of a dose response gradient (+ 1) |
Increasing level of evidence based on a dose-response gradient is rarely appropriate when assessing effectiveness of an intervention |
Residual confounding would further support inferences regarding treatment effect |
If some plausible confounders have not been documented, there is no credible way to determine how adjusting these parameters would alter the effectiveness estimates. |
Disagree with the criterion |
Increase with 1 level if all plausible confounders would reduce a demonstrated effect (+ 1) or would suggest a spurious effect if no effect was observed (+ 1). |
Increasing level of evidence is not appropriate because the confounders cannot be documented. Consequently there is a risk for biased conclusions |