Getting to grips with GRADE—perspective from a low-income setting

Case management of common illnesses has been a corner-stone of international strategies to reduce the unacceptably large number of childhood, newborn, and maternal deaths in low-income settings for more than 30 years [1-3]. The advent, more recently, of the global human immunodeficiency virus (HIV) pandemic and increasing international funding for tuberculosis, HIV, and malaria have reinforced efforts to develop international guidelines. In many low-income settings, the guidelines produced, usually emanating from or coordinated by multilateral health agencies, particularly the World Health Organization (WHO), have been accepted or endorsed after varying degrees of local adaptation. However, the adaptation process seems rarely to have received much scrutiny. For many years therefore international guidance, based on expert opinion and narrative accounts of the evidence [4], has been translated somewhat opaquely into national guidance at country level. In one sense, this was probably a relatively efficient mechanism through which to influence national policies. But it perhaps relied more on the respect countries had for WHO in particular and their limited technical capacity to offer an alternative rather than the quality of the underlying evidence.

Governments are however increasingly urged to develop evidence-based policies with support from their technical arms and advisers and in collaboration with the research community [5]. This translation of evidence into policy and subsequently practice is seen as an important contribution to improving health [6,7]. The greater availability of systematic reviews on clinical topics and especially the expansion of the Cochrane Library should in theory have made this process considerably easier. Although true to a degree for many topics of specific relevance to low-income settings, there are often inadequate or no trial data or no reviews. Reviews of work on the accuracy of clinical diagnosis are a particular gap. Many available reviews from high-income countries focus on therapy and assume that the patient population of relevance is defined by a competent clinician who has access to reliable diagnostic testing. Yet guidelines for many conditions in low-income settings are to be used by health workers with limited training and little or no access to diagnostic tests. They thus commonly begin not with the name of the condition (or diagnosis) affecting a patient to which the guideline should be applied but with a definition to apply to the general patient population presenting. These eligibility criteria are almost invariably based on the presence or absence of specific clinical features that should be present before empiric treatment is initiated. The ability of clinical criteria to identify treatment recipients is thus at least as important as the efficacy of the treatment [8,9].

If relevant reviews are not available to inform policy, then they need to be conducted. Unfortunately, the skills and resources required to conduct relevant reviews within low-income settings have remained relatively limited despite some efforts in this direction [10]. Paradoxically, therefore, improving the methodologies for evidence synthesis in support of guideline development from an expert-led, narrative process to a requirement for systematic reviews may have decreased the ability of many countries to develop guidelines independently, at least to international standards. The advent of Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) [11] has continued to raise the technical bar. An acceptable guideline should now be based on high quality systematic reviews of the evidence, with findings translated into evidence tables after considering the adequacy of the research studies and possible limitations of the data available, including applicability to the population or context (indirectness) and precision [12]. This treatment of the evidence should precede its formal consideration together with local values and the feasibility, risks and benefits, and costs of introducing a proposed new guideline recommendation [13]. Given the relative complexity of this entire process, its recent development and the prior and persistent lack of capacity in many low-income countries, the GRADE approach therefore risks putting apparently even greater distance between what is possible in such countries and what is desirable.

An obvious response from low-income settings might be that the GRADE approach is therefore simply not, at present, appropriate. This would be, in our opinion, a missed opportunity. Attempting to make the process of moving from evidence to recommendation structured and inclusive of local values and contexts promotes the active involvement of countries. To adapt or develop locally, evidence-informed policies emphasizes the need for important local data. The resulting process could result in greater ownership of, and transparency in, the guideline development process at a national level and help prioritize topics for local data collection or research. So how might we realize the advantages of approaches, such as GRADE while overcoming the problem that low-income settings could increasingly be found standing still while best practices accelerate away from them?

The clear resource and capacity limitations in low-income settings mean it would make little sense for every low-income country to develop guidelines de novo. In this regard, and perhaps unlike high-income settings, the long history of and experience with a largely centralized process of evidence summary and guideline development may be an advantage. Furthermore, many topics for which guidelines are required are common to multiple countries, whereas the volume of available research is limited and sparsely spread across them. There are potentially therefore considerable efficiency advantages in sharing the results of evidence synthesis and appraisal processes. In the immediate future WHO, having adopted the GRADE approach, has committed itself to undertake (or at least coordinate) much of the work related to evidence synthesis that should support international guidelines for common disorders. An immediate imperative therefore is that all products of such evidence synthesis and quality assessment (and any updates) should be made rapidly and freely available. A more technical consideration is that for many topics of relevance to low-income countries, the absence or limitations of available randomized controlled trial data mean that methods to incorporate evidence from observational studies are needed when conducting syntheses and deciding on its quality.

However, the efforts required to produce and make evidence summaries available in support of many guidelines relevant to low-income countries go well beyond the current roles of bodies, such as the Cochrane Collaboration and will require both strategic thinking and funding. Although WHO and partners may be taking the lead in the production of early GRADE evidence summaries, and be the obvious repository for them, low-income countries should increasingly be able to contribute. This will, however, demand deliberate efforts to explain, popularize, and build capacity in the preparation of evidence using systematic reviews and GRADE approaches. If the GRADE approach is to become more than the preserve of WHO or international bodies, efforts at capacity building need to begin soon, as even for completed summaries each country will perhaps need to consider for themselves the level of indirectness of the evidence.

Our initial experience in using GRADE to help revise pediatric guidelines in Kenya supports the desirability of efforts to share work on evidence summary and quality appraisal and identifies some challenges. Thus, using only informal contacts, we came across a second group with a then unpublished review, including GRADE summary tables, for 1 of the 12 key topics we were working on. Although there were, unusually in our experience to date, several randomized controlled trials relevant to the topic that both groups had identified and included, our syntheses and interpretation of these data differed. The difference hinged on how to pool the data from all the trials for a critical outcome, mortality, and reflected differing views on the nature of the clinical condition and defining clinical subpopulations (With few trials and endpoint events, statistical tests for heterogeneity are probably unhelpful). On the basis of this difference, one review concluded that there was a clear and major impact on mortality, whereas the other concluded that there was at best moderate quality evidence and a nonsignificant trend in favor of a mortality reduction. This difference in opinion is perhaps an inevitable consequence of the difficulty in standardizing decisions about the clinical populations, relevance, nature, indirectness, and quality of evidence demanded within GRADE. Yet, it does indicate the potential value of making work readily available, possibly even with registers of work in progress, to avoid duplication of effort, although perhaps we should avoid considering any available report as absolutely definitive.

Cooperation over the generation and sharing of evidence summaries has an obvious rationale and has been used to good effect, even with minimal funding, in international child health [14]. Yet almost by definition the GRADE approach demands that progression from evidence to recommendations should be conducted at country level. WHO or bodies claiming to represent the international arena will, by definition, not be able to consider or represent national level values or feasibility. Thus, the final process of developing recommendations is a country-level activity. In turn, this means the recommendations arrived at in any one country may not be generalizable. There may, however, be considerable benefit from mechanisms that allow the sharing of experiences developing recommendations to help inform what will inevitably be incremental progress in improving guidelines using the GRADE approach. To this end, we hope to share our experience using GRADE, linked to a second national child health evidence week [15], to develop Kenyan pediatric and neonatal guidelines later in the year.

Our initial experience suggests that wider adoption of the GRADE approach may enable low-income countries increasingly to become partners in the use of evidence and the guideline development process rather than just recipients of international guidance. To facilitate this, specific efforts will be needed to develop capacity. Furthermore, coordinated efforts will be needed to avoid duplication of effort, to promote methodological development, and to share results and experiences. All of these would benefit from strategic thinking and funding.