Assessing risk of bias in randomised clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge

TH Foto‐Werbung/Science Photo Library

Randomised clinical trials are often inadequately reported and may be inadequately conducted. Any associated biases could impact seriously on the findings and conclusion of a systematic review. Authors of systematic reviews thus need to assess the risk of bias in included randomised clinical trials. In this 20th Anniversary editorial, we look at the evolution of guidance on appraising studies included in Cochrane Reviews.

Assessing the methodological ‘quality’ of included trials was addressed from the earliest days of The Cochrane Collaboration, although the phrase ‘risk of bias’ came into use later. In 1994 one of the first editions of the Cochrane Collaboration Handbook recommended that reviewers should routinely assess the adequacy of allocation concealment, and that they could consider assessing blinding and attrition, based on a seminal empirical study by Schulz and colleagues. Over the next decade several Cochrane Review Groups developed different recommendations for assessing risk of bias. Of 50 Cochrane Review Groups surveyed in 2007, 41 recommended using specific trial characteristics to assess risk of bias and nine either recommended using a quality scale or made this optional. Most groups suggested assessing the randomisation procedure (including concealment of allocation), blinding, and attrition.

In 2008 the Cochrane risk of bias tool was released with Review Manager 5.0, following three years of development. It included six characteristics: ‘generation of the allocation sequence’, ‘concealment of the allocation sequence’, ‘blinding’, ‘incomplete outcome data’, ‘selective outcome reporting’, and ‘other bias’. Selective outcome reporting was included based on a landmark paper documenting a tendency for statistically significant trial outcomes to be selected for reporting. Subsequent research has replicated this finding. In 2011 a revised version of risk of bias tool split blinding into ‘blinding of participants and personnel’ and ‘blinding of outcome assessment’.

Now is a good time to reflect on two decades of assessing risk of bias. The risk of bias tool provides a standardised approach, based on items selected on both theoretical and empirical grounds, and following broad consultations with clinical research methodologists. Furthermore, a more appropriate terminology has been developed emphasising ‘risk of bias’ instead of ‘methodological quality’, and the initial approach mainly based on a single trial characteristic to bias has matured into a structured multidimensional approach.

The risk of bias tool is a comparatively recent development that still likely needs refinement. The modest inter‐rater agreement rates will hopefully be improved by modifications to the questions and enhanced training courses. However, authors of Cochrane Reviews tend to be reluctant in designating an overall risk of bias for each trial or outcome and also reluctant to incorporate the risk of bias assessment in analyses and conclusions. The next version of Review Manager, scheduled for release by the end of 2014, will enable authors to see the risk of bias table jointly with the forest plot, thus facilitating a cohesive interpretation of effects and risk of bias for each outcome.

However, risk of bias assessment has more fundamental challenges. Empirical analyses of bias in randomised trials typically rely on meta‐epidemiological studies. Such studies involve comparisons within several meta‐analyses of the estimated treatment effects in trials with the characteristic present (such as adequate allocation generation) and trials without the characteristic (such as inadequate or unclear allocation generation). The risk of confounding in such comparisons is pronounced, as compared trials may differ for other reasons, such as allocation concealment, type of outcome, blinding, and the trial's sample size. Furthermore, reporting inadequacies in trial publications is an additional concern. It thus remains an open question whether inadequate allocation generation is truly causally linked to bias or whether it is an indirect marker of other factors associated with bias.

To establish reliable causal relationships of bias in observational studies may be even more difficult than to establish reliable causal relations in epidemiology in general. The assessment of risk of bias is to a large extent based on common sense and theoretical considerations, with an empirical basis of observational studies with a considerable risk of confounding.

This highlights a peculiar circularity. Meta‐analysis of randomised clinical trials is the core methodology for reliable estimates of treatment effects, and is thus the core methodology for evidence‐based medicine. This is partly based on the reasonable view that randomised trials are more reliable than observational studies in assessing effects of health care interventions. Still, the empirical evidence underlying the assessment of risk of bias in trials – an assessment necessary for ensuring that biased trials do not lead to biased systematic reviews – is based on observational studies.

An increasing number of meta‐epidemiological studies report associations between a trial characteristic and exaggerated treatment effects: funding status, number of centres participating in a trial, early stopping of a trial, and developing country status. For many of these characteristics it is unclear whether they represent a unique bias, confounding, publication bias, spurious findings, or a combination of these and/or other unknown factors. It is, nonetheless, helpful to be aware of such associations, sometimes called meta‐bias.

Funding status is a major concern in randomised trials. The exaggerated effects reported for industry trials may to some extent be explained as a result of publication bias or other characteristics included in the risk of bias tool, for example selective outcome reporting. However, companies that stand to gain financially by a positive result have substantial conflicts of interest when they control the planning, funding, conduct, and reporting of a trial. It is not clear that the risk of bias tool in its present version addresses this problem adequately.

It is important to assess risk of bias in randomised clinical trials included in systematic reviews. In the last 20 years risk of bias assessment in Cochrane Reviews has been refined several times. For the next two decades and beyond the process is likely to continue. The why is easy, the how is a challenge.

Feedback on this editorial and proposals for future editorials are welcome.

End Notes

Declarations of interest

The authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request) and declares (1) no receipt of payment or support in kind for any aspect of the article; (2) no financial relationships with any entities that have an interest related to the submitted work; and (3) that DM is a member of The Cochrane Library Oversight Committee, but no other relationships or activities that could be perceived as having influenced, or giving the appearance of potentially influencing, what was written in the submitted work.