Box 2.

: Some examples of problems with methods

• Inclusion and exclusion criteria were not well defined,^w31,w51 inconsistently transferred from title to methods section (another category of patients or another type of intervention),^w9,w31 or applied inconsistently,^w31 subjectively,^w29 or with retrospective rationalisation.^w27 The problems partly related to choice of terminology—for example, “experimental or quasi-experimental designs” (meaning randomised or quasi-randomised or non-randomised studies?) resulting in different interpretations or enumerations in the “selection criteria,” “description of studies,” “main results,” meta-analytic graphs, and “table of included studies” and in conflicting numbers^w31,w51
• Concealment of allocation was mixed up with double blinding,^{w4,w20,w29,w31} or discrepancies existed between the quality of the trials as described in the table of included trials, the graphs, and the main text.^w31,w52 These problems may occur much more often than we found as most occurrences were noted by a single assessor
• Assessors were particularly concerned about loss to follow up when rates of dropout were high (29%, 43%, 50%),^w29,w51 dropouts were treatment failures,^w29 or the reviewers claimed to have performed an intention to treat analysis but the numbers on the graphs did not fit with the number of included patients^w52; the assessors asked for subgroup analyses including only trials with full follow up, more thorough documentation and discussion of loss to follow up,^w4 or more cautious conclusions^w29,w51
• Outcome measures were inappropriately combined (death plus surrogate outcome),^w53 inappropriately split (different lengths of follow up),^w20 too numerous without precautions against multiple testing,^w31,w52 unblinded subjective assessments,^w53 or derived from only a single small trial^w31
• Statistical problems related to extremely different standard deviations between experimental and control group^w18 or between trials^w31 or to a confidence interval with zero length^w51