Risk of Bias 2 in Cochrane Reviews: a phased approach for the introduction of new methodology

Our confidence in the results of research studies is affected by how they are designed, conducted and reported. In ideal circumstances, all studies would accurately determine the true effect of a health care intervention. However, studies do not take place in an ideal world, and flaws or problems in the way they are conducted could lead to them either overestimating or underestimating the true effect of the intervention. Understanding the potential flaws or problems in studies included in systematic reviews is important as they can distort or bias the results, which means the findings might not accurately represent the truth. Assessing the design, conduct and reporting of studies included in Cochrane Reviews is therefore an important step in understanding whether there is any risk of bias in the results, supporting our confidence in the review's conclusions.

Since 2008, authors of Cochrane Reviews have been expected to use the Cochrane risk of bias tool to assess the risk of bias in any randomized trials that they identify, to help users understand the trustworthiness of the review findings. Formal evaluations of user practice over the last decade revealed a need to improve the tool. For example, some bias domains were considered challenging for users (particularly incomplete outcome data and selective outcome reporting), the tool was frequently implemented in non‐standard ways (e.g. with important domains omitted or inappropriate domains added), and the tool was not well suited to cluster randomized trials and crossover trials. To address these limitations and to reflect current understanding of how the causes of bias can influence study results, a new version of the tool, Risk of Bias 2 (RoB 2), was launched in 2019 and included in version 6 of the Cochrane Handbook of Systematic Reviews of Interventions.

The revised tool is structured into five domains of bias, according to the stages of a trial in which problems may arise: (1) the randomization process; (2) deviations from intended intervention; (3) missing outcome data; (4) measurement of the outcome; and (5) selection of the reported result. The judgement for each domain is ‘low risk of bias’, ‘some concerns’, or ‘high risk of bias’. In addition, the same three judgement options are available for overall risk of bias. The assessments apply to a specific result from the trial, rather than to the study as a whole. An important innovation is to specify whether the result being assessed is being interpreted as the effect of assigning participants to interventions (the intention‐to‐treat effect) or to the effect of participants adhering to their assigned intervention according to the trial protocol (the per‐protocol effect). The implications of deviations from the intended intervention are different between these, particularly for trials in which participants are aware of their intervention.

The implementation of RoB 2 has been a strategic priority for Cochrane and was included in the Strategy to 2020 (www.cochrane.org/about-us/strategy-to-2020) as part of the goal to develop and continue to produce high‐quality, relevant, up‐to‐date systematic reviews. The first Cochrane Review that uses RoB 2 has recently been published. It follows new guidance for reporting RoB 2 assessments in Protocols and Reviews, and illustrates new designs for interactive, results‐level tables for these assessments.

The review by Williams and colleagues examined the effects of physical activity interventions for people with congenital heart disease. The RoB 2 tool was used to examine results of the included trials in relation to five outcomes: maximal cardiorespiratory fitness; submaximal cardiorespiratory fitness; health‐related quality of life; physical activity; and muscular strength. The risks of bias were judged by the review team to be similar across these outcomes and in most cases the results were rated overall as having ‘some concerns’. The review team stated that trialists had not reported well the information they needed to judge bias in measurement of the outcome (Domain 4), which includes questions concerning blinding of outcome assessors. Nor had most trials presented sufficient detail in any pre‐specified statistical analysis plans to allow assessment of bias in selection of the reported result (Domain 5). Health‐related quality of life was, however, judged to be at high risk of bias across all randomized trials that reported it: it was self‐reported with a lack of blinding of outcome assessors, and this was thought to have the potential to affect how it was reported. This led to an assessment of ‘very low’ certainty in the health‐related quality of life outcome using GRADE, which was downgraded twice due to risk of bias (also referred to as study limitations). Other outcomes in the review were not downgraded for risk of bias.

Introducing new methods and tools for Cochrane Reviews is not without its challenges. Cochrane has 52 Cochrane Review Groups (CRGs), with editors, staff, and authors who need to be trained and supported in building the confidence and expertise within their Groups. Updates to technology are generally needed, including amendments to Cochrane‐owned technology (such as RevMan) as well as to those with which we collaborate and that authors frequently use, including Covidence, DistillerSR, and GRADEPro. Other updates to guidance, tools, and training are also required, along with plans to ensure the method or tool is being used appropriately. For RoB 2, we decided that a phased implementation approach would be used, beginning with a pilot phase period during which authors and CRGs may opt in to use the tool (methods.cochrane.org/news/implementation-risk-bias-2-cochrane). The aims were for a supported and gradual roll‐out of RoB 2 in Cochrane Reviews to observe common issues as they arose and to put training and support in place before scaling up to wider implementation.

Led by members of the Cochrane Editorial and Methods Department (EF and KD), the pilot phase is helping us understand and overcome obstacles, gather evidence on the usability of the tool, and understand the impact it has on technology, publishing, and presentation. Fundamentally, the pilot is helping us work towards a streamlined and efficient process for authors and editors, from writing the protocol to publishing the full systematic review using the RoB 2 tool (community.cochrane.org/news/what-you-need-know-about-risk-bias-2-rob-2-cochrane). As of 23 October 2020, there were over 80 Cochrane Reviews across all eight of the CRG Networks using or planning to use RoB 2, with the number steadily increasing over time.

As part of the pilot phase, the Cochrane Methods Support Unit (methods.cochrane.org/about-us/cochrane-central-executive-methods-team/methods-support-unit) will work with CRGs to ensure RoB 2 is being used accurately in protocols and reviews. As well as providing feedback directly on protocols and reviews, the Methods Support Unit are coaching editors and staff on what to look for, helping build methodological expertise within the CRGs. Updates on implementation and access to key resources for using RoB 2 in Cochrane Reviews, such as an introductory leaflet, starter pack, and FAQs, can be found on the Cochrane Methods website (methods.cochrane.org/risk-bias-2). Hosting in‐person and virtual training has also been important, including a RoB 2 Cochrane Learning Live Webinar Series (training.cochrane.org/rob-2-learning-live-webinar-series). The pilot phase allows the Editorial and Methods Department and the Bias Methods Group to understand gaps in guidance and support in real time so that we can ensure they are addressed as soon as possible.

RoB 2 provides a more appropriate way to assess risk of bias in randomized trials and should help review authors draw more appropriate conclusions about the included evidence. The phased implementation of RoB 2 is ensuring we understand what challenges the improved tool may pose to authors and editors so that the necessary guidance and infrastructure can be put in place to alleviate them. Review teams who plan to use RoB 2 in a Cochrane Review are encouraged to contact the Cochrane Methods team (methods@cochrane.org) so that they can benefit from additional guidance and support.

Acknowledgements

RoB 2 research group. Core group: Julian Higgins, Jelena Savović, Matthew Page, Asbjørn Hróbjartsson, Isabelle Boutron, Barney Reeves, Roy Elbers, Jonathan Sterne. Working Group members: Doug Altman, Natalie Blencowe, Mike Campbell, Christopher Cates, Rachel Churchill, Mark Corbett, Nicky Cullum, Francois Curtin, Amy Drahota, Sandra Eldridge, Jonathan Emberson, Bruno Giraudeau, Jeremy Grimshaw, Sharea Ijaz, Miguel Hernán, Sally Hopewell, Asbjørn Hróbjartsson, Peter Jüni, Jamie Kirkham, Toby Lasserson, Tianjing Li, Stephen Senn, Sasha Shepperd, Ian Shrier, Nandi Siegfried, Lesley Stewart, Kate Tilling, Ian White, Penny Whiting. And: Henning Keinke Andersen, Vincent Cheng, Mike Clarke, Jon Deeks, Daniela Junqueira, Alexandra McAleenan, Geraldine Macdonald, Richard Morris, Mona Nasser, Nishith Patel, Jani Ruotsalainen, Holger Schünemann, Jayne Tierney

The authors would like to thank Kayleigh Kew for her comments on an early draft of this Editorial.

Feedback on this editorial and proposals for future editorials are welcome.

End Notes

Declarations of interest

EF, KD and TM are employed and receive a salary from Cochrane. EF and KD are leading the RoB 2 implementation project 2 in Cochrane, and TM is heavily involved in the project. JH and MP are Convenors of the Cochrane Bias Methods Group. JH is one of the lead developers of the Risk of Bias 2 tool.

Provenance and peer review

This editorial was commissioned based on a proposal by the Cochrane RoB research group, and it was not externally peer‐reviewed.