The Role of The Cochrane Collaboration in Support of the WHO Nutrition Guidelines^,

Abstract

This article describes the background and contribution of The Cochrane Collaboration to the WHO Nutrition Guidelines program. Systematic reviews, augmented by the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) methodology for assessing the quality of a body of evidence, form the evidence basis for WHO guidelines. Our shared experience of working together has highlighted a number of issues that are challenging, such as decisions made about selecting appropriate questions for evidence synthesis and the nature of study types that are included, in particular the decision on whether or not to extend a search beyond randomized studies. Although the skills and experience required for evidence synthesis are different from those needed to determine recommendations for policy and practice, our experience suggests that some engagement between the two groups is mutually beneficial. Finally, our experience highlights the recognition that evidence of effectiveness is essential but by no means sufficient to guide decisions on recommendations. Programmatic and implementation considerations are important to guide decision making and the evidence basis for this may be limited; therefore, it is essential that groups involved in delivering interventions to populations are also engaged in the guidelines process.

Introduction

In a landmark article in The Lancet in 2007, Oxman et al. (1) demonstrated that despite internal guidance to the contrary, and against the wishes of some of those involved, WHO guidelines were not being developed on an evidence-informed basis. The WHO moved decisively to address this deficiency to the effect that all public health recommendations published from 2010 are required to demonstrate compliance with the WHO’s Handbook for Guideline Development (2). This states that “WHO recommendations need to be based on the best evidence available . . . an effective approach is to perform a systematic review using specific questions about the intervention(s) likely to be recommended in the guideline” (3).

Systematic reviews have been defined as “Scientific investigations that focus on a specific question and use explicit, pre-specified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. (They) may include a quantitative synthesis (meta-analysis), depending on the available data” (4). The key elements of a systematic review are as follows:

• Identification of a clear and specific question, including, for effectiveness reviews, a description of the relevant Population, Intervention, Comparison(s), and Outcomes of interest (PICO)

• Prespecification, via a review protocol, of the proposed methodologic approach, including the search strategy, study selection criteria, and approach to data analysis of included studies

• A comprehensive and systematic search for relevant primary studies

• Selection of relevant studies using the predefined criteria and process

• Critical appraisal of included studies for risk of bias

• Extraction of data from included studies and data analysis following prespecified and transparent methods

• Synthesis of the characteristics and results of the included studies and meta-analysis where appropriate

• Presentation and interpretation of the findings of the review

When appropriately conducted and reported, systematic reviews can provide the best available evidence by virtue of the transparency, inclusiveness, and rigorous approach. By reducing selective reporting, and by accumulating all of the relevant evidence, systematic reviews can reduce the uncertainty and increase the precision of pooled results.

For an increasing number of guidelines bodies, including the WHO, a vital ingredient and a bridge between the evidence synthesis in the systematic review and the recommendations made by a guidelines panel is the production of an evidence profile and summary of findings table that uses the methodology developed by the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE)³ working group (3).

In this article I describe the role of The Cochrane Collaboration in developing systematic reviews and GRADE evidence profiles in the development of guidelines for the WHO Nutrition Guidelines program, and some of the lessons learned in this process.

The Role of Systematic Reviews in the Development of Guidelines

Most reputable national and international guidelines groups now routinely use systematic reviews as the evidence basis for their guidelines. The association between systematic reviews and guidelines was exemplified in the work undertaken by the U.S. Institute of Medicine to produce the following two reports, launched at the same event in May 2011: “Finding What Works in Healthcare: Standards for Systematic Reviews” (4) and “Clinical Practice Guidelines We Can Trust” (5). The Institute of Medicine reports presented a vision of the relation between the systematic review producer and guidelines development group, each with distinct attributes and skills but both essential to the process. Crucial to this is an appreciation that evidence of effectiveness is only one factor to be considered by guidelines groups in determining whether to recommend one intervention over another: cost, feasibility, the challenges of implementation via programs, as well as the values and preferences of populations are also central to evidence-informed decision making. This is a theme that was particularly relevant for the work undertaken by the WHO Nutrition Guidelines group and The Cochrane Collaboration.

What Is The Cochrane Collaboration?

The Cochrane Collaboration is celebrating its 20th anniversary in 2013. The movement was initiated as a response to the British epidemiologist Archie Cochrane (6), who noted in 1979 that “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.”

The Cochrane Collaboration was formally launched at the first Cochrane Colloquium in Oxford, United Kingdom, in October 1993 and attended by >70 researchers from across the world. The movement now comprises some 31,000 dedicated people based in >120 countries. Its mission statement is “to promote evidence informed health decision making by producing high quality, relevant and up-to-date synthesized research evidence.” The principal and most well-known manifestations of this industry are Cochrane Reviews, which are published as the Cochrane Database of Systematic Reviews (CDSR) as one database of The Cochrane Library. Fifty-three Cochrane Review Groups (CRGs), each responsible for an area of health, work with contributors (review authors) to produce Cochrane Reviews using a consistent and assiduous process that begins with acceptance of a review title; through development, peer review, and publication of a protocol; to development of the completed, peer-reviewed systematic review. To facilitate the process, the Collaboration has developed its own bespoke software application Review Manager (RevMan), which must be used to produce all reviews that are published in the CDSR. A unique feature of Cochrane Reviews is that from the outset, the intention is to maintain reviews, with review authors committing to update the reviews, usually on a 2-y cycle.

The growth in the number of Cochrane Reviews is shown in Figure 1. There are currently >5000 active reviews in the CDSR, and in each year ~450 new reviews and slightly more updated reviews are published. Access to the reviews is generally via a national, regional, or institutional license, although people living in countries covered by the Hinari schemes enjoy “1-click” access to The Cochrane Library (7). In addition, as a consequence of the Collaboration‘s contribution to the WHO Nutrition work, all nutrition-related reviews are now fully accessible via the WHO’s e-Library of Evidence for Nutrition Actions (eLENA) portal (8). In 2012, Cochrane Reviews were visited in full text >5.4 million times.

FIGURE 1.

Growth in the number of Cochrane Systematic Reviews. Reproduced from The Cochrane Collaboration with permission.

Various CRGs have contributed to WHO guidelines work previously, but the relation with the Nutrition Guidelines began in February 2010 after a conversation between J. P. Pena-Rosas, then the Coordinator of the WHO Micronutrients Unit, and Cochrane’s Editor in Chief (the author). As a consequence, the Cochrane Editorial Unit and 5 CRGs were commissioned to complete or update 12 reviews relating to vitamin A in advance of the guidelines process relating to the vitamin based on questions in the PICO format identified by a panel of experts identified by the WHO micronutrients group (9–20). Since the work on vitamin A was completed, the Cochrane Editorial Unit has also contributed to work on other guidelines programs, including those on iron.

What Forms of Evidence (i.e., Randomized Controlled Trials vs. Everything Else)?

For evaluating effectiveness of health interventions, the randomized controlled trial (RCT) is seen as the gold standard by most authorities (21), and following Archie Cochrane’s dictum, it is not surprising that the Collaboration has concentrated mainly on studying RCTs as the primary source of high-quality evidence. Cochrane has sometimes been characterized as being overly dominated by RCTs, although some CRGs, including the influential Effective Practice and Organization of Care Review Group, have for many years routinely searched beyond this type of study. However, it is widely acknowledged that, in some circumstances, RCTs provide insufficient information to guide practice, and that for evaluating evidence of harms, and in particular, severe, rare, and delayed harms, observational studies are required (22, 23). Equally, for different types of research questions, exploring diagnosis or prognosis, for example, different hierarchies exist.

The Use of GRADE in Systematic Reviews

In common with many guidelines producers, the WHO uses the GRADE methods to assess the quality of a body of evidence (24). This addresses a key aspect of the review findings: how confident are we that our estimate of the effectiveness of a given intervention lies close to the actual effect? This is crucially important if we are not to place undue emphasis on the direction or magnitude of any reported effect without any consideration of the extent to which we can be confident that this accurately reflects the true effect. The GRADE system differs from other quality-grading systems in a number of key areas:

• Grading of the quality of a body of evidence is ordered by outcome rather than by study. This is much more meaningful to most readers of reviews who are primarily interested not in the underlying studies but in what they are collectively able to say about the effect on the preferred outcomes (e.g., survival, symptom control, etc.).

• Grading is not wholly determined by the nature of the study. In other systems, quality is rigidly determined by this, but using GRADE it is possible both to downgrade the quality rating from RCTs in particular circumstances but also to upgrade evidence from observational studies.

• Grading decisions are transparent, so that the reader can independently judge whether they are justified.

The GRADE approach provides 4 easily understandable ratings: high, moderate, low, and very low. High-quality evidence indicates that the review authors are very confident that the true effect estimate lies close to the estimated treatment effect. For moderate, low, and very low quality evidence, this confidence is reduced incrementally and the likelihood that the two are substantially different increases.

Randomized studies start with a high grading but can be downgraded if the studies that reported on the outcome in question had a high risk of bias (internal validity), indirectness, inconsistency of results, or imprecision, or where there is evidence of reporting bias. Indirectness, sometimes referred to as a measure of external validity, may relate to different populations; the nature, dose, or route of administration of the intervention or comparison; or the outcomes of interest. For each question, or comparison, no more than 7 outcomes are prespecified, and for each of these an assessment of the quality of evidence is made, alongside the results of the review. These can be presented as a summary of findings table or evidence profile. The GRADE working group has recently published a series of articles in the Journal of Clinical Epidemiology that provide a more detailed description of the GRADE methods (22).

What Have We Learned during the Process and How Could This Inform Future Work?

This work has provided us with many opportunities for learning and reflection. The process starts with identification of the questions of interest: the PICO criteria. In our experience, the panel of experts drew up a list of PICOs that exceeded the expectations, and differed somewhat in nature, from those identified by the systematic reviewers. Both sides can learn from this, but it was clear that as the populations of interest, interventions, and comparisons became more granular, the likelihood of finding definitive evidence to guide recommendations was reduced. Two other aspects of the question setting included the nature of nutrition interventions, which clearly differ from other health care interventions in some respects, as well as the issue of non-clinical outcomes. In all of the reviews and guidelines in the program, recommendations were often based on laboratory measurements, and evidence on mortality and morbidity outcomes was frequently sparse.

Second, there was the important question of study selection. Unsurprisingly, the WHO panel, in common with many such question-setting panels, was keen to explore nonrandomized studies and in particular to identify and evaluate these where the evidence from RCTs for important outcomes was insufficient. However, in some cases, the systematic review teams were unwilling, or unable, to accommodate this. This is an area that remains hotly disputed, with many in the review synthesis world holding to the view that nonrandomized studies can rarely, if ever, provide definitive and wholly valid estimations of the magnitude of an effect, and can introduce unpredictable and misleading bias (20).

Third, we found that it was very useful to have representatives of the systematic review team present during the guidelines committee deliberations. In some cases, this enabled the reviewer to explain certain decisions, but also, on occasion, the committee asked for the review to be modified, for example, to exclude a very old trial. This is evidently a potential source of bias, but on balance we felt that the requests were reasonable and it was an advantage to be able to recalculate the data in “real time” during the meeting.

However, perhaps the main learning point that became increasingly clear as the guidelines process continued was that the evidence of effectiveness was only one consideration among many in the guidelines process. In particular, it was striking that in all cases, the organizational and programmatic considerations needed to be fully explored and considered before decisions on recommendations could be made. Other articles in this series will develop these issues in more detail, building on the experience and expertise of their authors, but the importance of recognizing this cannot be exaggerated (25, 26).

Conclusions

In summary, our experience of the WHO guidelines panel working with members of The Cochrane Collaboration has demonstrated the importance of systematic reviews within the guidelines process. The use of the GRADE method is a useful and intuitive bridge between the complexities of evidence synthesis and the recommendations made by the guidelines committee. There will continue to be debate on what forms of evidence provide the most valid and useful evidence. This should neither undermine the importance of randomization for addressing effectiveness questions nor hide the reality that, for different questions, different hierarchies of questions exist, and for some harms, randomized studies are clearly poorly suited to the task.

However, what the process demonstrates most vividly is that whereas high-quality evidence is essential to the process, it is by no means sufficient. High-quality evidence on implementation would be desirable, but in practice, the experience of people working in programs on the ground must be incorporated into the process. How best to achieve this remains a challenge for all parties.