Application of the GRADE Approach in the Development of Guidelines and Recommendations in Genomic Medicine

Abstract

A great deal of ambiguity exists in the development of guidelines for genomic applications used in clinical practice. The GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach has the potential to be applied in the guidelines and recommendations development process in genomics. Here, we discuss whether and how GRADE can be applied to address the challenges posed by the evidence-based guidelines and recommendations development process in genomics. To see how GRADE can complement to the current guidelines development in genomics, we compare and contrast GRADE with other approaches. GRADE differed from other methods by incorporating patient values and preferences and balance of consequences. We conclude that the groups trying to implement genomics into practice may gleam more information from applying the GRADE framework. However, it is not clear yet whether GRADE can address the issue of timeliness in terms of the differences between the time required for guidelines development and the rapid pace of genomics.

Background

There is a great deal of ambiguity around how guidelines and recommendations development groups review evidence and develop guidelines, defined as recommendations for practitioners about the care of patients with specific conditions based on best available evidence and practice experience, for genomic applications.¹ The lack of systematic approaches to develop clinical practice guidelines for genomic medicine is emerging as a challenge for the research community. The need to develop guidelines and recommendations for genomic medicine has been commonly agreed on by almost all major genetic societies, bioethics committees and organizations in their position statements, policies, and recommendations. Like in any other discipline of medical sciences, guidelines and recommendations for genomic medicine are essential to ensure the appropriate use of genomic applications and avoid any potential risks and complications associated with their misuse. For the purpose of this opinion article, we define genomic medicine as defined by the National Human Genome Research Institute, i.e. “an emerging medical discipline that involves using genomic information about an individual as part of their clinical care (e.g. for diagnostic or therapeutic decision-making) and the health outcomes and policy implications of that clinical use.”²

The GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach has been introduced as a system of rating the quality of evidence for guidelines development that reportedly offers a transparent and structured process for carrying out the steps involved in developing recommendations. Developed by the GRADE Working Group, the GRADE approach is a systematic approach to make decisions about the quality of evidence and strength of recommendations. More specifically, the approach evaluates methodologic rigorousness of the studies included in the guidelines development process, consistency of results across the studies, and generalizability of the results to wider patient base.³ Many national and international organizations have adopted the GRADE approach for evidence-based guideline development.⁴

It has been reported that the GRADE approach provides a useful framework for grading both the quality of the evidence behind a recommendation and considering how strong the recommendation should be.^5,6 The GRADE approach may overcome the limitations of the previous guidelines development systems and has been adopted by more than 70 organizations around the world including World Health Organization, the Cochrane Collaboration, National Institute for Health and Care Excellence, Centers for Disease Control and Prevention (CDC) Advisory Committee on Immunization Practices, and Scottish Intercollegiate Guidelines Network.⁷ The GRADE approach starts by asking a relevant question according to the Population, Intervention, Comparison, and Outcome (PICO) criteria, gathers the best available evidence to answer the question, assesses the quality of evidence, and evaluates the trade-off between risks and benefits keeping patients’ perspectives before making the recommendations.⁸

The GRADE approach has been successfully applied in several fields. The approach was applied in diagnostic tests in the process of developing clinical practice guidelines for the diagnosis of cow’s milk allergy⁹ and managing severe sepsis.¹⁰ In the latter example, the GRADE grid instrument was developed by the Surviving Sepsis Campaign whereby consensus could be attained among the guideline developer in cases of uncertainty surrounding the interpretation of scientific evidence.¹⁰

The approach has also been applied in the guidelines development process for genomic applications such as prenatal genetic testing,¹¹ genetic testing for patients at risk of Lynch syndrome,¹² venous thromboembolism,¹³ and hereditary hemochromatosis¹⁴ which are discussed in depth later in this article.

Some authors have concluded that GRADE approach to grading the quality of evidence and strength of recommendations for diagnostic tests provides a comprehensive and transparent approach.¹⁵ However, further discussions are needed to reach a common consensus if and how the GRADE approach can be applied for guidelines and recommendations development in genomic medicine.

Several groups have come up with approaches to develop guidelines for genomic applications such as Discretionary Advisory Committee on Heritable Disorders in Newborns and Children (DACHDNC), American College of Medical Genetics and Genomics (ACMG), Office of Public Health Genomics (OPHG), CDC, Dutch Pharmacogenetics Working Group (DPWG), and National Society of Genetic Counselors (NSGC)—Practice Guidelines Committee. Owing much to the fact that no single guidelines development group entirely covers the genomics field except Evaluation of Genomic Application in Practice and Prevention (EGAPP) that assesses evidence for genetic tests, CDC OPHG, and the National Cancer Institute’s Division of Cancer Control and Population Sciences held a workshop in March 2013 to discuss approaches to knowledge synthesis and evidence-based guidelines development in genomic medicine.¹ It was proposed in this workshop that due to paradigm shift in genomic medicine, more robust and agile methods and approaches are required to generate evidence and develop guidelines in this field. The workshop was mainly focused on comparing the existing guidelines development approaches with the Institute of Medicine’s (IOM) report “Clinical Practice Guidelines We Can Trust,” and it was argued that the guidelines development process in genomic medicine is unlikely to be entirely consistent with IOM recommendations.¹

Discussion

Further building on the discussion held by Schully et al, we take forward this discussion and argue that if a broad-based consensus can be reached in case of the GRADE approach instead of the IOM recommendations and if this approach could possibly serve the purpose of guidelines and recommendations development process in genomic medicine. The main purpose of this opinion article is to reflect on the fundamental question, if and how the GRADE approach can address the challenges posed by the evidence-based guidelines and recommendations development process in genomic applications. In the following sections, this question is further dissected to discuss the application of the GRADE approach within the context of genomic medicine.

Why there is a need for systematic approaches to develop guidelines and recommendations in genomic medicine?

Efforts are being made to develop and test approaches to the development of guidelines and recommendations in genomic medicine. Several organizations/groups have developed approaches to guidelines development for genomic applications. These organizations/groups include DACHDNC, ACMG, OPHG, DPWG, and NSGC, among others.

However, the current grading systems lack clear guidance on how to evaluate, link, and make recommendations on the basis of different bodies of evidence.¹⁶ Furthermore, these efforts are hindered by the mismatch between the length of time required for guidelines development and the fast pace with which the field of genomic medicine is moving. This creates an inherent problem to the process as the evidence typically is outdated before the guidelines are ready and out.¹ Even though the traditional approaches to guidelines and recommendations development are being practiced, more systematic but rapid approaches need to be explored.¹⁷ This challenges the research community to develop, implement, and validate guidelines and recommendations for genomic medicine that are nimble and can keep up with the field.¹

Is the GRADE approach a robust, systematic, and comprehensive approach for guidelines and recommendations development?

The GRADE approach is a comprehensive and structured way to rate the quality of evidence in systematic reviews or other synthesized evidence. All the steps involved in the approach are interconnected and not necessarily sequential. The GRADE approach starts by defining the question in terms of the PICO criteria and proceeds with a systematic search to identify all the available evidence on the subject matter. The quality of the evidence is rated based on 5 factors (risk of bias, inconsistency, indirectness, imprecision, and publication bias) that may downgrade and 3 factors (large magnitude effect, dose response, and effect of plausible confounding factors) that may upgrade the quality of evidence. Next, all the information from the evidence synthesis is reviewed and a decision is made about the importance and criticality of the outcomes based on the recommendations being formulated, and overall quality of evidence is assigned based on the assessment. Finally, recommendations are formulated with the direction (for or against) and strength (strong or weak) of the recommendations.¹⁸ Figure 1 provides an overview of the GRADE approach.

Figure 1.

Overview of the GRADE approach. GRADE indicates Grading of Recommendations Assessment, Development and Evaluation.

The GRADE approach may be regarded as one of the highly recommended approaches to guidelines development and has been described as a comprehensive, explicit, and transparent methodology for grading the quality of evidence and strength of recommendations.¹⁹ More than 100 grading systems have been reported in the literature,²⁰ and the GRADE approach has emerged as a centralized and transparent method to translate evidence-based medicine to guidelines.²¹

There have been disagreements on whether the GRADE approach is superior to other approaches. A comparison article of the existing approaches to developing recommendations concluded that the GRADE approach is the most flexible methodology in terms of evaluating the evidence²² and is considered superior to other systems when it comes to the translation of evidence into recommendations. However, another review article comparing methods including both GRADE and EGAPP among other several approaches ranked EGAPP above the GRADE approach in terms of methodologic and process characteristics.¹⁶ Another empirical study to review the current experiences with the GRADE approach concluded that the approach, currently being used by several public health organizations, is applicable to public health, and that it was well-received by the groups who applied it in terms of a systematic, transparent, and rigorous process.⁴

One of the key strengths of the GRADE approach is that it can be applied regardless of the fact whether the quality of the relevant evidence is high or low. It is the first system to evaluate the evidence by making difference between weightage of a poorly done randomized controlled trial (RCT) and a well-done RCT. For example, an RCT will not be regarded as a well-done RCT if executed with poor allocation concealment, high attrition, serious risk of bias, serious inconsistency between studies, serious imprecision, and likely publication biases.²³ However, if there is large effect size or a dose response gradient is shown, the RCT will be regarded as a well-done RCT.²³ However, the GRADE approach does not eliminate judgments or disagreements about evidence and recommendations but its merit is that it makes the whole process transparent.¹⁹

What are the major differences in the GRADE approach and other guideline and recommendations development approaches?

Some of the major differences between the GRADE approach and other approaches to guidelines and recommendations development have been described as the ability of the GRADE approach to (1) evaluate relative importance of the outcomes of interest, (2) differentiate clearly between quality of the evidence and the strength of the recommendation, (3) provide explicit criteria to measure the quality of evidence regardless of the study design, and (4) propose a structured and specific process for developing recommendations.⁷

This difference in evaluating the quality of evidence and strength of recommendations can be explained by the example of comparing EGAPP approach with that of the GRADE system. The EGAPP approach was established to develop a systematic process for assessing the validity and utility of genetic tests in clinical practice. The EGAPP approach strongly relies on study designs during the assessment of the quality of evidence, potentially conflating assessment of the quality with the strength of recommendations, whereas the GRADE approach focuses on overall strength of evidence in a much broader and comprehensive manner, such as described above, in determining the strength of recommendations.²⁴ For example, in the process of developing recommendations for diagnostic tests, the GRADE approach assigns a “high-quality” rating to valid observational studies of diagnostic accuracy and then proceeds further to systematically identify factors that might lower the rating of these studies.²⁴

Is the GRADE approach suitable for guidelines and recommendations development in genomic medicine?

Among the existing approaches to guidelines development, EGAPP, DACHDNC, ACMG, OPHG, DPWG, and NSGC guidelines have been developed to address the challenges of guidelines development in genomic medicine. However, some of the major challenges faced in genomic medicine may still be a hindrance for these guidelines development approaches to be comprehensive. For example, in genomic medicine, there is a lack of empirical studies of direct comparison between the interventions under consideration. In such cases, the GRADE approach may provide a confidence gradient by providing the strength of recommendations by 4 factors, ie, the risk-benefit balance, quality of evidence, patient values and preferences, and costs and resource utilization.^6,25 Given the fact that RCTs are not always possible in most of the disciplines of genomic medicine, a strong argument for using the GRADE approach in genomic medicine may be that with the application of this comprehensive, robust, and transparent system, even studies traditionally considered to be of low importance in evidence generation, eg, case reports and other uncontrolled clinical observations, could lead to strong recommendations, whereas high-quality evidence such as RCTs do not necessarily have to lead to strong recommendations.²⁶

For instance, it has been strongly argued that the GRADE approach can even be used in cases where diagnostic intervention studies, such as RCTs and observational studies comparing the impact of alternative diagnostic strategies, are not available such as in most of the cases in the emerging field of genomic medicine. In such cases, valid studies of test accuracy also start as high quality in the GRADE system.¹⁵

In genomic medicine, other than quality of evidence, variation in preferences and values, resource consumption, ethical, legal, and social implications affect the strength of recommendations.²⁷ Considering such factors in developing guidelines and recommendations in genomic medicine necessitates the use of more comprehensive and systematic approaches.²⁴ The GRADE approach may provide a good example of such comprehensive and systematic approaches in the process of guidelines and recommendations development because some of the above factors, i.e. variation in preferences and values, and health care resource consumption are well taken into consideration in the GRADE approach. It is important how patients perceive the value of performing certain genetic testing even though a definitive treatment may not be available for the specific condition under investigation. For example, performing the genetic testing for Huntington chorea may be beneficial if it reduces anxiety in patients or if confirmation of diagnosis improves patient’s well-being through provision of prognostic information,¹⁵ even though there would not be an effective treatment in this particular case.

Potentially, the GRADE approach can be applied in the guidelines and recommendations development process in genomic medicine. There is a growing support in using the GRADE approach in the development of guidelines and recommendations in genomics, e.g. NSGC supports the use of GRADE approach in assessing and evaluating the body of evidence.²⁸ There are already cases in genomic medicine, where the GRADE approach has been applied, e.g. the use of the GRADE approach in developing guidelines in the Health Evidence Review Commission’s (HERC) recommendations on “Prenatal Genetic Testing” on hemoglobinopathies, cystic fibrosis, Tay-Sachs disease, and aneuploidy screening.¹¹ The commission (HERC) is an entity that reviews medical evidence to prioritize health care spending and promotes evidence-based medical practice in the state of Oregon in the United States.²⁹ The commission used 4 criteria, i.e. balance between desirable and undesirable effects, quality of evidence, resource allocation, and values and preferences, guided by the GRADE approach to develop their recommendations.

To understand how the GRADE approach was applied by HERC, we illustrate the application of the GRADE approach to cystic fibrosis as shown in Figure 2. Characterized by early onset of intestinal malabsorption, failure to grow, and recurrent chest infections, cystic fibrosis can lead to death in early childhood if not treated. The relevant question in this case was whether prenatal genetic screening should be conducted to identify mutations in the cystic fibrosis transmembrane conductance regulator gene or not. The HERC synthesized evidence from population-based studies, reviews, and existing guidelines. The National Institute for Health and Care Excellence in United Kingdom, for instance, does not recommend carrier screening test for cystic fibrosis. However, with the GRADE approach, HERC found sufficient evidence to support the use of prenatal screening for cystic fibrosis if the results can inform decision making in childbearing or fetal diagnosis. The screening can be beneficial to couples who are at risk and can be offered genetic counseling and prenatal diagnosis. It was found that there were potential benefits of the screening with minimal harm. After assessing the available evidence, and balancing the desirable and undesirable consequences, HERC recommends prenatal screening for cystic fibrosis once in a lifetime (weak recommendation).

Figure 2.

Application of the GRADE approach to cystic fibrosis recommendations development by HERC. GRADE indicates Grading of Recommendations Assessment, Development and Evaluation; HERC, Health Evidence Review Commission.

The US Multi-Society Task Force on Colorectal Cancer also used GRADE approach to develop guidelines for the genetic testing of patients at risk of Lynch syndrome.¹² The GRADE approach was applied in conjunction with the National Cancer Institute’s guidelines for cancer genetic studies.

In another study, GRADE approach was applied to evaluate outcomes of genetic testing in persons with history of venous thromboembolism. The approach was used to grade the evidence whether factor V Leiden testing alone or in combination with prothrombin G20210A testing leads to improved clinical outcomes. The study concluded that there was no direct evidence to support that the genetic testing to test for mutations was leading to any improved outcomes.¹³ It is interesting to note that the study was conducted by the Johns Hopkins University Evidence-Based Practice Center at the request of EGAPP, and they decided to choose the GRADE approach for the evidence grading in this particular case.

Finally, the American Association for the Study of Liver Diseases used the GRADE approach to develop guidelines for the diagnosis of hereditary hemochromatosis through HFE (hemochromatosis gene) mutation analysis.¹⁴

Many argue that the current approaches to the assessment and grading of recommendation should be changed to reflect some of the aspects of the GRADE approach because it focuses on the overall strength of evidence for each outcome in the process. On behalf of the EGAPP working group to review and update the EGAPP approach, Veenstra et al²⁴ reported that although EGAPP has not made a decision to update recommendation language to make it consistent with that of the GRADE approach, refining and adopting EGAPP methods to the GRADE approach would make the former more comparable with other contemporary methods. However, the National Academies of Science, Engineering, and Medicine (NASEM) reported recently that EGAPP has drawn on methods used by the GRADE approach in their evaluation of evidence for genetic tests.³⁰

Both GRADE and EGAPP are evidence-based approaches but their components are different yielding different answers. Table 1 shows the major differences between the GRADE and EGAPP approaches in terms of inclusion and consideration of key elements in the guidelines development process.

Table 1.

Key differences between the GRADE and EGAPP approaches.

Elements in guidelines development	GRADE approach	EGAPP approach
Selecting the topic/framing the questions	☑	☑
Quality of supporting evidence	☑	☑
Resource use (cost-effectiveness)	☑	☑
Patient values and preferences	☑	☒
Balance of desirable and undesirable consequences	☑	☒

Abbreviations: EGAPP, Evaluation of Genomic Application in Practice and Prevention; GRADE, Grading of Recommendations Assessment, Development and Evaluation.

To see how GRADE can potentially complement to the current guidelines development approaches in genomics, we compare and contrast GRADE with EGAPP approach by demonstrating the example of type 2 diabetes (T2D)—EGAPP’s insufficient evidence label:

1. Even though an element of the EGAPP approach, cost-effectiveness was not included in the T2D example, whereas consideration of the resource use is an integral part of the GRADE approach. It is argued for in the GRADE approach that it is prudent that guideline panels consider and document estimates of resource use because a diagnostic or therapeutic intervention may increase or decrease its use compared with the alternative intervention which may have different implications in different settings and for patients with different socio-economic status and may quickly change over time.

2. Patient values and preferences were not mentioned in the EGAPP’s T2D example. In the GRADE approach, it is stressed on that the guideline panels judging the importance of outcomes must ensure that their decisions reflect patients’ values and preferences (patients’ liberty and autonomy). Patients’ view about what constitutes benefit or harm, and clinicians’ understanding of particular outcomes for patients can differ.

3. There was not much emphasis on desirable (improved quality of life, reduced morbidity, longer survival, and less resource use) versus undesirable (complications from procedures and medication, consequences of incorrect diagnosis, adverse effects such as increased morbidity, burden, and higher resource) consequences in EGAPP’s T2D example. However, the GRADE approach emphasizes both on the desirable and undesirable consequences.

4. The 2 approaches do not significantly differ in grading the quality of evidence, in terms of weighing RCTs and observational studies.

5. Both EGAPP and GRADE approach do not regard “expert opinion” as evidence in the guidelines development process. However, it is emphasized in the GRADE approach that even though “expert opinion” is not a category of evidence but is nearly always necessary to integrate and contextualize evidence (either from a clinical or methodologic point of view).

In theory, it is probable that by applying the GRADE approach to the EGAPP’s insufficient labeled T2D example, the results would have likely changed if patient values and preferences and quality of life were considered in the guidelines development process. However, it is worth mentioning that such a change in results would be entirely based on patient values, preferences, and quality of life, given that the 2 approaches do not differ in grading the quality of evidence.

The incorporation of these additional elements in the EGAPP approach is of paramount importance because EGAPP is also used as a tool for the health technology assessment (HTA) of genomic technologies. Among 7 organizations (Blue Cross Blue Shield Technology Evaluation Center, Emergency Care Research Institute, EGAPP, Hayes, Institute of Clinical and Economic Research, US Preventive Services Task Force, and UpToDate) that adopt the HTA framework in personalized medicine, EGAPP was found to be the only organization focused solely on genomics.³¹ The NASEM recently reported that EGAPP’s strengths are its flexibility and customization in evaluation of various topics in genetics. However, the report pointed to a potential weakness of EGAPP approach stating that the approach is only focused on single-gene tests and its application to broader genomics may be difficult.³⁰

What future steps are required to integrate the GRADE approach into genomic medicine for guidelines and recommendations development?

The challenges associated with the process of evidence generation around genomic medicine have always been the topic of priority discussions, and up till this time, there still exists some major gaps in the integration of genomic medicine into public health. Owing to the urgency of enhancing high-quality guidelines and recommendations development in genomics,¹ and the fact that there are no “universal principles” shared by the organizations involved in the evaluation of genomic applications,^16,32 there is a need that robust measures are taken to integrate the GRADE approach into genomic medicine for guidelines and recommendations development. Further steps are required to actively involve all the relevant stakeholders in an action-oriented dialogue to assess the feasibility and appropriateness of the GRADE approach for evidence-based guidelines and recommendations development in genomic medicine.

Such action-oriented dialogues should be complemented and well informed by preceding rigorous knowledge synthesis measures such as systematic literature reviews and in-depth interviews with the relevant stakeholders, to assess the feasibility and implementation of the GRADE approach in genomic medicine, to develop guidelines and recommendations, and design strategies to integrate the approach in this rapidly developing field. In cases where consensus is elusive, e.g. developing guidelines for direct-to-consumer genetic tests or genetic tests, the GRADE approach can be complemented with other techniques to enhance the accuracy of the guideline and recommendations development process.¹⁰

Conclusions

We argue that groups trying to implement genomics into practice may gleam more information from applying the GRADE approach. The EGAPP approach, being used as a key tool of HTA in genomics, should envisage the relevant elements from the GRADE approach. However, it is not clear yet whether the GRADE approach can address the issue of timeliness in terms of the differences between the time required for guidelines development and the rapid pace of genomic medicine. Like many other guidelines development approaches, the GRADE approach is an evidence-based approach but with different components yielding different answers. The intended outputs and preferences of organizations applying these approaches determine which approach to take.