Grading nutrition evidence: where to go from here?

See corresponding article on page 1578.

Systematic reviews and meta-analyses are useful tools in summarizing a large body of evidence and informing policy and guidelines. Although findings from systematic reviews and meta-analyses are regarded by some to be the most authoritative form of available evidence, there is great potential for the misuse of the systematic review and meta-analysis methodology (1). The prevalence and magnitude of such misuses in systematic reviews are particularly concerning in nutrition research owing to the proliferation of meta-analyses in the nutrition literature and increased reliance on using systematic reviews to develop food policies and dietary guidelines (2). In their study published in The American Journal of Clinical Nutrition, Zeraatkar et al. (3) at McMaster University conducted a comprehensive review and analysis of 150 randomly sampled systematic reviews of nutritional epidemiology studies published between January 2018 and August 2019. Their detailed quality assessment highlighted several common methodological problems in published meta-analyses of nutritional epidemiologic studies.

One of the methodological issues raised by the study was that only a small proportion of the 150 systematic reviews (10%) implemented a formal evaluation of the certainty of the evidence, and “most did not discuss risk of bias.” A careful assessment of certainty of evidence and risk of bias in systematic reviews is critical to evaluate the quality of overall evidence on specific nutrition topics and these are, therefore, important considerations in developing clinical and public health guidelines. To this end, Zeraatkar et al. recommended the GRADE (Grading of Recommendations Assessment, Development and Evaluation) system for rating the certainty of a body of evidence (4). The GRADE system, which was originally developed to evaluate the quality of clinical intervention evidence, relies on a hierarchy of study designs: meta-evidence from randomized trials is automatically considered to be high certainty, whereas meta-evidence from nonrandomized studies is regarded as low certainty owing to potential confounding and selection bias (5). Although the GRADE system is relatively straightforward to implement in assessing the strength of the evidence from randomized double-blinded and placebo-controlled trials, the interrater reliability of GRADE ratings for complex interventions that are not amendable to double-blinding or placebo-controls is only modest (6). The reliability of the GRADE system to evaluate the strength of observational evidence is likely to be more uncertain given the heterogeneity of observational study designs and different degrees of exposure measurement errors and adjustment for confounding factors. Although we agree with the authors that it is important to “[maintain] consistent standards for evaluating the certainty of evidence across health fields,” the complexity of environmental and behavioral exposures such as diet warrants additional considerations when grading the evidence, and one should not blindly apply the existing GRADE criteria to the development of public health guidelines regarding diet, lifestyle, and environmental factors.

Recently, a series of systematic reviews rated the meta-evidence for the relation between intake of red and processed meats and risk of major chronic disease incidence and mortality as “very low and/or low certainty” using GRADE, and consequently, the authors recommended individuals to continue their red and processed meat consumption habits. These recommendations have caused a great deal of public confusion (7) and raised doubt about the appropriateness of using the GRADE system in developing nutrition recommendations (8). A separate research group has proposed a modified system for rating the certainty of meta-evidence from nutritional studies (NutriGrade). Although NutriGrade shares several scoring components with the GRADE criteria, it does not automatically consider the evidence from observational studies as low certainty. Instead, the assessment of evidence certainty is based on an overall quantitative score of 9 components. Applying NutriGrade to the same body of meta-evidence on red meat intake and chronic disease risk resulted in ratings of “moderate quality” and “high quality” on the associations of red and processed meat intakes with mortality (9) and type 2 diabetes (10), respectively.

The red meat example underscores the challenge in assessing the quality of meta-evidence for diet and lifestyle factors (11). Using the current GRADE criteria, meta-evidence from observational studies (without clear distinction between their different types, e.g., cohort studies compared with case-control studies) is all rated low certainty owing to lack of randomization, which can be further downgraded to very low level of certainty for additional reasons (e.g., because of unknown confounders and significant heterogeneity) (5, 12). To overcome the problem of excessive downgrading of observational evidence, ROBINS-I (Risk of bias in non-randomized studies of interventions) was developed to assess the risk of bias across multiple domains, rather than simply the lack of randomization, in grading the certainty of observational evidence (13). Although the integration of ROBINS-I with GRADE may provide a more balanced approach to grade observational evidence, the validity of this complex system has not been carefully assessed in the context of nutritional, lifestyle, and environmental exposures.

Methodological problems in assessing the risk of bias in nutrition research are not limited to observational studies. Currently available risk of bias instruments including GRADE often fail to capture common limitations of dietary intervention trials including poor dietary adherence and high dropout rates. In addition, because most dietary interventions are focused on food substitutions while maintaining the same total energy intake, the effects of interventions are likely to vary with the types of replacement foods. For example, the effects of red meat consumption on cardiovascular disease risk may depend on whether red meat is replaced by plant-based protein foods such as legumes and nuts or starchy foods such as bread and potatoes. The current GRADE system, heavily relying on the clinical intervention paradigm, does not adequately consider these methodological issues when assessing the strength of evidence from dietary intervention studies.

Another issue in applications of GRADE to nutrition research is that dietary interventions are seldom designed to test the effect of the exposure on a hard clinical endpoint, but often rely on intermediate outcomes (e.g., fasting glucose, blood lipids, and blood pressure) as surrogate endpoints. Whereas the Bradford Hill criteria consider experimental evidence from small and mechanistic studies on intermediate endpoints when evaluating causality (14), the GRADE system does not consider the issue of biological plausibility as a domain of certainty. Another important consideration in the Bradford Hill criteria is a dose–response relation from observational studies, which can estimate the effect of dietary exposures across the full distribution of dietary intake of a population to evaluate dose–response and nonlinear relations, whereas dietary intervention studies typically test a fixed dose of dietary exposure.

For these reasons, there is a critical need for the GRADE system and similar metrics to be modified to accommodate unique characteristics of nutritional epidemiology and intervention studies. So where to go from here? The first step is to improve the overall rigor and quality of systematic reviews and meta-analyses by adhering to a set of guidelines as discussed by Zeraatkar et al. including a priori registration of systematic review protocols, stricter adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, and careful assessment of heterogeneity and biases. We should be cognizant that although systematic reviews and meta-analyses are valuable, a meta-analysis is only as good as the studies it pools together. Too often, systematic reviews of nutrition topics are conducted by authors who do not fully appreciate the complexity of the subject or lack content knowledge, which can lead to flawed methodology and misleading conclusions. Moreover, it is important to recognize the inherent limitations of systematic reviews and consider the conclusions in the broader context of the field (1).

Second, we need to be cautious in applying existing tools to grade the quality or certainty of nutritional evidence. Although the GRADE system was initially developed to assess the strength of evidence from clinical interventions, it has been increasingly used to evaluate the evidence for lifestyle and environmental exposures. However, the infeasibility of conducting large long-term randomized trials for most dietary and lifestyle factors renders the current GRADE criteria inadequate for these exposures. Recent developments such as integrating ROBINS-I for bias assessment with the GRADE system may improve its applicability for nutrition research, but further validation of this combined approach is clearly warranted. Alternative approaches such as those of the World Cancer Research Fund (15), the Hierarchies of Evidence Applied to Lifestyle Medicine (HEALM) (16), and NutriGrade (12) should also be carefully evaluated. Interestingly, a recent systematic review applied GRADE and NutriGrade to the same body of meta-evidence for low-carbohydrate dietary intervention trials and type 2 diabetes remission and found that the 2 metrics concurred for only 53% of the study outcomes with respect to the certainty level of the evidence (17). Considering the pros and cons of different grading systems and the complexity of nutritional studies, there is an urgent need for the nutrition science community to come together and develop a consensus on the appropriate tools for nutrition evidence synthesis and grading.

Notes

This work is supported by NIH grants HL60712 and DK 46200 (to FBH).