Comment on “Perspective: NutriGrade: A Scoring System to Assess and Judge the Meta-Evidence of Randomized Controlled Trials and Cohort Studies in Nutrition Research”

Dear Editor:

We read the Perspective article “NutriGrade: A Scoring System to Assess and Judge the Meta-Evidence of Randomized Controlled Trials and Cohort Studies in Nutrition Research,” which was published in November 2016 in this Journal (1). We agree it is important to assess the trustworthiness of evidence.

The authors describe the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach as not being applicable to nutrition and nutrition research and suggest a scoring system to overcome this. We are not convinced that this is the right approach and would encourage collaboration in a joint approach with GRADE rather than setting up something separate. GRADE is a community of scientists, physicians, and public health specialists that has been in existence for >17 y, aiming for a common approach across diverse topics. GRADE aimed to clarify the confusion that has led to the development of nearly 100 grading systems without a clear rationale for doing so (2). The GRADE approach has been endorsed and adopted by >100 international organizations and societies, which cover a wide variety of clinical, public health, and methods areas. Although the author's tool based on its name may be perceived as endorsed by the GRADE Working Group, the contrary is the case. Indeed, some aspects of the suggested tool even contradict the conceptual underpinnings of the GRADE approach (3, 4). What we aim to do with this letter is provide some background to GRADE and encourage collaboration and harmonization, which is a fundamental strength of the GRADE approach and the function of the GRADE Working Group.

GRADE is a common and transparent approach to grading certainty (or “quality”) of evidence and strength of recommendations. It was developed over more than a decade by the GRADE Working Group (www.gradeworkinggroup.org), consisting of >500 members with different expertise and with involvement of numerous international organizations. GRADE constantly refines and develops its methods and extends its reach through global dialogue and careful, transparent scientific consensus development. For example, there are currently project groups working on GRADE to assess the certainty of evidence in systematic reviews on environmental toxins, qualitative research synthesis, values and preferences, and animal translation models. Each group works within defined frameworks of “project groups,” with careful refinement of the methods until these are finalized, approved by the GRADE Working Group, and published. GRADE is open to newcomers and established researchers alike.

In the field of nutrition, GRADE has been applied successfully as part of Cochrane and non-Cochrane systematic reviews (5–7). For example, 118 of 470 nutrition-related Cochrane Reviews published in 2015 used GRADE to assess the certainty of evidence (8). Nevertheless, the authors do refer to “several limitations” that arise when applying GRADE; however, it is not clear to us what limitations the authors are actually referring to. For example, lack of blinded randomized controlled trials and the resulting sparse bodies of randomized evidence is not a methodologic shortcoming of the GRADE approach but a limitation of the evidence base. In addition, this issue is not unique to nutrition but applies to other fields such as rare diseases and surgical interventions. Furthermore, GRADE does not classify systematic reviews, but rather, the certainty of bodies of evidence obtained through systematic reviews or other appropriate forms of evidence synthesis.

In terms of the authors’ suggestions about the advancements with their scoring system, we would question their appropriateness and validity. The authors are not convincing in their argument as to why randomization would not be critical to balance known and unknown prognostic factors in nutritional studies (9). There is no plausible rationale or supporting evidence to justify their approach to include funding bias as a separate item. In terms of conflict of interest, GRADE captures financial and nonfinancial interests through the existing domains for risk of bias (in particular, selective outcome reporting), indirectness, and publication bias (10). In addition, algorithmic scoring approaches for the assessment of “quality” are inferior given that they imply inevitably assigning “weights” to different items in the scale, and it is difficult to consistently justify the weights assigned (11). We encourage the authors of this article and interested readers to further explore how GRADE works and to join in advancing the methods in a unified approach.