Methods of Formal Consensus in Classification/Diagnostic Criteria and Guideline Development

Abstract

Guideline or diagnostic criteria in clinical practice assist physicians in their clinical decision-making and improve health outcomes for patients. Diagnostic and classification criteria are based on evidence from rigorously conducted controlled studies. Formal group consensus methods have been developed to organize subjective judgments and to synthesize them with the available evidence. This review discusses four types of formal consensus methods used in the health field and their applications in rheumatology: the Delphi method, nominal group technique, RAND/UCLA Appropriateness Method, and National Institutes of Health consensus development conference.

Physicians are faced with diagnostic and treatment dilemmas on a daily basis in their clinic setting. Well developed diagnostic or treatment guidelines in clinical practice summarize data for physicians to improve their clinical decision-making for an individual patient. (1). Evidence-based-guidelines derived from rigorously conducted controlled studies outperform expert opinion; however, in practice there are instances where sufficient research-based evidence does not exist (2). Various group consensus methods incorporate opinions of a group of experts rather than an individual expert in a formalized manner (3, 4). Given the diversity of opinion that can occur with the diagnosis and management of disease, formal group consensus methods organize subjective judgments and to supplement the available evidence. Formal group consensus methods allow for inclusion of a wide range of knowledge and experience, interaction between members, stimulate constructive debate, and prevent influential behavior of one opinion to formulate suggestions about a specific question where there is insufficient evidence. Since perfect agreement is seldom reached, the objective of the consensus methodology is to identify a central tendency among the group and grade the level of agreement reached. Consensus methods are increasingly being used to develop diagnostic, classification, therapeutic guidelines and response criteria sets in various fields of medicine including rheumatology (5, 6).

Our review will focus on details of the consensus methodologies along with the relative advantages and disadvantages of each method. There are other formal quantitative methods to combine information from the literature such as meta-analysis and systematic review and that the purpose of this review is to concentrate on consensus when other quantitative methods are not possible either due to paucity or conflict in the literature.

Defining consensus

Consensus does not need to be defined as full agreement among participants (7). A pre-specified range is decided by the group running or leading the consensus methodology, according to the needs of the task to be performed. There are several methods for defining consensus with options including overall consensus of the group and consensus on each topic being considered by the group (8).

• A final vote determining percent agreement (e.g., 80%) amongst participants

• Rating scale- a specified mean rating must be achieved on each topic for inclusion by group

• A majority of participants must give a topic a certain rating for inclusion of a topic (as detailed later in the RAND-UCLA methodology)

Participants

The composition of the group is important in determining the decision reached. Most agree that a participant should be an expert and have credibility in the appropriate field (8, 9). An expert can be a clinician with vast clinical expertise, a researcher who is well versed with the current literature, or a lay person or patient who has experienced the impact of the disease or intervention or condition in question (4, 10). Ideal characteristics of a group participant would be researchers and clinicians with expertise in the field providing face validity and facilitate dissemination of consensus meeting results (8). In addition, the participants should be enthusiastic about the project and understand the demands and responsibilities of it.

Heterogeneity of the participants (different levels of expertise within a field, different fields of expertise, different geographical areas and population ethnicity or different health care policies in different countries) has advantages and disadvantages to the consensus process. A diverse group may lead to better performance than homogeneity in terms of considering all relevant areas of the topic (11) but may also lead to disagreement among group participants due to different diagnostic and therapeutic approaches based on geographical area. Also, a participant with higher authority may exert a greater degree of influence on the group when consensus group is comprised of people of diverse status, whereas consensus derived by a more uniform group will be more likely to reflect the majority view (12). In our view, heterogeneity of the participants brings more credibility to the consensus methodology and a more robust, creative product is developed.

In addition, larger groups will increase the reliability of the judgment (13), but may be difficult to coordinate and more time consuming. In general, reliability of consensus recommendations declines rapidly when the group size falls below 6, and above 12, improvement in reliability is not substantial (13). An exception is the Delphi and nominal group techniques that can recruit larger number of participants.

Importance of Evidence from Literature when using Group Consensus

Although group consensus participants are generally recruited on the basis that they have superior knowledge of the published literature in the field, it is essential to supplement participants with up-to-date, evidence based literature. When literature reviews are provided, the evidence is used both in the initial discussion and in later decision making (14). A literature review (preferably a systematic review) also provides a common starting point for the group and increases the perception that the process should be as evidence based as possible (10). The amount of information varies from no information because available evidence is weak to comprehensive synthesis of relevant research when there is abundant information in literature but no general consensus (8). Any level of information, however small is likely to influence the group decision (12), and thus should be provided.

In the development of clinical guidelines for care, incorporating an evidence base has been clearly shown to result in better guidelines. In a retrospective study comparing sets of guidelines, six sets of breast cancer guidelines were classified as evidence based, consensus based, or a combination of both. Instruments to measure quality of guidelines were used to grade the sets of guidelines. The evidence based guidelines scored highest on quality whereas strictly consensus based guidelines scored the lowest (15).

Formal Consensus Methods

The focus will be on four types of formal consensus methods used in the health field: Delphi method, Nominal Group Technique, RAND/UCLA Appropriateness Method (RAM), and National Institutes of Health (NIH) consensus development conference methodology (Table 1). In practice a combination of two formal consensus methods or their modifications can be used in a two step process, where one method is used for item generation or some initial consensus and then other method is used for final consensus. When there has not been rigid adherence to one consensus methodology, the description of the procedure in the methods section is referred to as a “modified” consensus method.

Table 1.

Characteristics of informal and formal consensus development methods (20).

Consensus development method	Mailed* questionnaires/ Survey	Private decisions elicited prior to group discussion?	Formal feedback of group choice	Face- to-face contact	Interaction structured	Incorporates evidence	Aggregation method
Delphi	Yes	Yes	Yes	No	Yes	+	Explicit
NGT	No	Yes	Yes	Yes	Yes	+	Explicit
RAND	Yes	Yes	Yes	Yes	Yes	+++	Explicit
Consensus dev. Conf.	No	No	No	Yes	No	+	Implicit

^*postal survey, internet, fax

Delphi Methods or Technique

The Delphi method, originated in the 1960s, takes its name from the Delphic oracle's skills of interpretation and foresight (16). It was developed at the RAND Corporation to obtain the most reliable consensus of opinion of a group of experts on a subject in a systematic manner (17, 18). It attempts to do this by series of well-defined questionnaires based on surveys and feedback. Usually Delphi is applied when consensus among large number participants are needed who cannot be grouped into a single room for logistic or economic reasons (19).

Process (Figure 1)

Figure 1.

Schematic diagram of Delphi process step by step.

The first step is to define a specific problem or question to be solved by the Delphi process (20). The purpose for the Delphi should be made clear. For example, in case of criteria set development it is important to make it explicit that the goal of the Delphi process is to develop diagnostic/ classification criteria set for the specific clinical entity.

Prior to Round 1

The team (or Steering Committee) undertaking the Delphi invites clinical or research experts to provide opinions on a specific matter and participate in subsequent questionnaire rounds. The panel of participants must be adequate (can be hundreds, but at least 10–30) and should represent multi-disciplinary leaders from various geographic areas on the particular subject in consideration. Participants should be motivated and interested in the problem to be solved, which will help ensure adequate response rates to the survey.

Sometimes a thorough literature search is carried out by the team undertaking the Delphi process to study the existing evidence, which may help to define the first set of questions. The results of the literature search and the relevant original publications are sent to the participant to facilitate the process. Given the privacy issues it has now becoming standard first to send an invitation letter to the group for participation in the Delphi process. This initial mail helps establish relationship, verify e-mail address, and provide the denominator that will be used to calculate the response rate For example, in an ongoing effort to develop outcome measures for connective tissue disease-interstitial lung disease clinical trials, appropriate expert panelists were identified via an international investigation of specialist associations, study investigators of multicenter endpoint studies in parenchymal lung disease, members of respective consensus committees and task forces, dedicated to parenchymal and rheumatologic lung disease, members of disease specific groups as well as authors of key publications, in journals and textbooks. Patients with connective tissue disease-interstitial lung disease are also included.

Round 1

The participants are asked broad open-ended questions on the Delphi topic to elicit different opinions. These questions are usually more descriptive to elicit the expert’s opinions, novel ideas, and their experience on a specific sub-heading/question. The opinions gathered are grouped together under a limited number of headings and statements drafted for circulation to all participants in a questionnaire form. This initial survey should be as simple as possible and is ideally no longer than a page.

Researchers can send detailed instructions of the process and ask the participants to provide group and items relevant to the project. Alternately, they can propose certain groups (e.g., organ systems and patient reported outcome measures as done recently in scleroderma (21) relevant to the study and have the participants propose additional groups and items.

Round 2

The replies from the participants in round 1 are analyzed with simple descriptive statistics and summarized to generate a series of statements. At this point there may be consensus on some issues. Those issues not reaching consensus are developed into a focused second questionnaire and sent back to the individual participants. Feedback from round 1 is also reported to the participants. Depending upon the question, the participants either answers focused questions in yes/no manner to reach immediate consensus or rank their agreement or disagreement with each statement (or items) in the second round. The questions should be clearly and concisely stated and response elicited should be in a simple and straightforward fashion.

Round 3

In the second survey, the panelists are provided with the overall results and his/her previous reply/scores. Experts then reconsider their previous opinion and re-rate their level of agreement with each statement in the questionnaire, with the opportunity to change their score in view of the group's response. The re-ratings are summarized and assessed for degree of consensus: if an acceptable degree of pre-defined consensus (which should be predefined before initiation of the Delphi) or when sufficient information exchange has been obtained is obtained, the process may cease, with final results fed back to participants; if not, the third round is repeated. Ideally consensus should be reached by round 3 but in theory many more rounds can be done.

Researchers have used a pre-defined cut of the mean/ median values (21) or done cluster analysis to discriminate important from unimportant items (22).

In summary, the process is conducted usually over 2 to 4 "rounds," with the results elicited, tabulated, and reported to the group after each round.

Delphi Technique

Advantages

Disadvantages

Large number of participants possible (23) Each participant expresses their opinion freely and impersonally Personal contact between experts Limits dominance by eminent, eloquent, or highly opinionated individuals in the field Less likely that the moderator of the panel may bias the group Substantial amount of time to express ideas, reflect upon answers and make changes Cheap, convenient and no geographical constraints Easy to understand, flexible, and can be applied to broad range of topics Can be used preceding NGT meeting for initial item generation

Generalizability of the study findings (external validity) Dependent on questionnaire design Vulnerability with respect to who is an "expert" Obliviousness to reliability measurement and scientific validation of findings (24) Potential for bias exists in participant selection Consensus panel judgments influenced by panel composition and by feedback given during the panel process (25) Coordinating large groups and several rounds can be complicated and costly Delphi does not allow any personal contact between the experts

Examples in Rheumatology

Delphi strategies have been used to solve an array of problems in health and medicine, from the needs of an individual hospital or department to those of a statewide agency or state (23, 24). Delphi technique has been also widely used in the past in the internal medicine as well as rheumatology literature to develop diagnosis and management guidelines. Specifically, in the rheumatology literature (25–27), Delphi has been used for criteria set development especially for reaching consensus on the outcome and response criteria set in diseases like systemic sclerosis, systemic lupus erythematosus, gout, ankylosing spondylitis and psoriatic arthritis. Use of Delphi exercise in classification criteria set development has been limited and is mostly used in initial phase to gather data and opinion in an open-ended and systematic manner or to make initial consensus on broad categories of variables which are further narrowed down by other consensus methods like NGT in the second part of the consensus development process (5, 6, 28–30).

Recommendations for use

Delphi exercise is best used for reaching consensus on variables to be used in outcome or response criteria set.

Nominal Group Technique

Nominal Group Technique (NGT) is derived from social-psychological studies of decision conferences and management science studies. It is a face-to-face structured group meeting of experts that is led by an experienced moderator. (8). NGT was historically used in the social services, government, education, and industry since the 1960s.

In NGT, a question is generated for which an expert panel is convened. The expert panel may consist of 5–9 panelists or could be larger. Larger groups can be separated into several groups (or tables) of 5–9 participants which work simultaneously on the same questions (31).

In brief, NGT asks for silent and independent generation of ideas, the round-robin listing the ideas, serial discussion led by a moderator, and independent ranking of the ideas. NGT is preferably done in a room with rectangular table arranged in an open “U” with flip chart or large screen computer at the open end of the table. Each panelist is asked to generate ideas to specific questions and record privately on a piece of paper for 5–10 minutes. In the second phase, in round robin fashion, participants are asked to provide each of their ideas which are listed on a flip chart or wide screened computer visible to the entire group. No discussion is conducted at this time. In the third phase, a serial brief discussion is led by the moderator; the goal of the discussion is clarification of the ideas or statements (31). During the last phase, each idea is privately ranked or rated on a scale of 1–5 or 1–10. The highest ranking solutions will be kept while the remaining solutions are discarded. Since the NGT can be influenced by strong personalities, the process requires that at each round the first person to speak is different from the previous one; this means that the first round will start with the first person to the left of the moderator, the second round with the second person to the left of the moderator etc. In this way all people will have the possibility to speak first and avoid the unduly influence of strong personalities (19). There are no guidelines regarding the cut-off for what constitutes a high vs. low ranking. Generally 70– 80% consensus is required (32) although researchers may chose to have a lower cut-off but this should be pre-defined (21).

It is important to remember that the purpose of NGT is to establish a prioritization of ideas and issues, and the use of numerical voting can assist in this (33). However, the temptation to attach greater meaning to the numbers, and carry out more sophisticated quantitative analysis should be resisted.

There are several variations of this process. Meetings can occur in several rounds with initial responses collected via mail (Delphi technique) followed by a future face-to-face-meeting. After a discussion period, additional rounds of generating ideas or re-ranking responses can occur. As used with RAND/UCLA method (detailed later), synthesized evidence can be provided to the expert panel prior to submitting solutions for the question that has been generated.

Nominal Group Technique

Advantages	Disadvantages
Participants meet face-to-face All participants have an opportunity to voice opinions Personal contact between experts Design of NGT does not allow any individual to dominate Group voting can occur if desired in later rounds	Certain members of the panel can take over discussion and drive results-experienced moderator required Limited by time— only a few questions can be discussed and agreed upon Economic and time costs associated with face-to-face meeting Limited to providing a solution to a few problems limits its applicability to multiple scenarios

Examples in Rheumatology

While often used in combination with Delphi methodology, there are some classification criteria sets developed with NGT for classification of childhood vasculitis and juvenile systemic sclerosis (5, 6). Other pediatric rheumatologic conditions for which criteria have been developed also include juvenile SLE (34, 35), juvenile idiopathic arthritis (36, 37) and inflammatory myositis (29, 38, 39). Combination of Delphi and NGT have also been used in development of provisional core set items (21). Other examples of NGT used in rheumatology include developing criteria for clinical outcomes in rheumatoid arthritis with the Outcome Measures in Rheumatoid Arthritis Clinical Trials (OMERACT) committee (40, 41) as well as development of the British Isles Lupus Assessment Group's disease activity index (BILAG) (42).

Recommendations for use

1. Primary use is to be give priority to questions/ items to be discussed. It can be adapted to be used for other circumstances such as

   1. Developing classification and response criteria.
   2. Developing consensus practice guidelines.
   3. Developing ideas for exploratory research (43)

This method is often used in combination with the Delphi method as described above. Pure NGT is useful when solutions need to be developed at a face-to-face meeting and expert opinion is needed more than evidence which may be lacking for the particular topic.

RAND-UCLA Appropriateness Method (RAM)

This was a method of group consensus developed in the 1980s by RAND (research and development) Corporation and UCLA (University of California-Los Angeles) (44). Using current scientific evidence in conjunction with expert opinion, this consensus method was initially developed to evaluate the overuse/underuse of medical or surgical procedures. This procedure has been used in the United States for assessing appropriateness of procedures such as coronary angiography, carotid endarterectomy, hysterectomy, and upper gastrointestinal tract endoscopy (45–47).

This process usually involves two interdependent groups: a core panel and an expert panel. The core panel guides the expert panel through the tasks of the RAM and provides synthesized data to the expert panel and the expert panel uses the data provided by the core panel to come to a consensus.

Prior to the consensus process, the core panel will conduct a systematic literature review with evidence synthesis to provide the expert panel with all pertinent information that will guide evidence-based decision-making (48). Then, a list of clinical scenarios (indications) is developed to provide the expert panel. These clinical scenarios will describe a patient with a set of characteristic features and the expert panel will be given a Likert-scale (typically 9-point) that will ask whether a particular intervention is appropriate for the patient.

The expert panel was historically restricted to 9 panelists but can range from 7–15 panelists (44). In general, an odd number is preferred to prevent a “tie.” It is recommended that a multi-disciplinary panel is selected so that bias from a like-minded group with identical agenda is avoided (49).

There are two rounds of rating appropriateness of an intervention. In the first round, the expert panel receives the clinical scenarios by mail (or via. internet as discussed in the Delphi section) and are asked to rate the appropriateness (on a 1 to 9 Likert scale) of an intervention. In RAM, rating between 1–3 is considered “inappropriate” (risks outweigh benefits), 7–9 is considered “appropriate” (benefits outweigh risks), and 4–6 is “uncertain”. They are asked to do this independent of the other panelists. They are allowed to use the synthesized evidence provided by the core panel overseeing the consensus process. The panel is asked not to consider the cost of performing the procedures or intervention in rating of their scenarios. There are other definitions proposed by RAM but this works best for a panel of 7–15 members and most widely used.

The second round is conducted as a face-to-face meeting over 1–2 days and is led by an experienced moderator. Usually 7–11 panelists are sufficient and it is recommended to have an odd number so there is no “tie.” All panel experts are given results of other expert’s individual ratings for all clinical scenarios. Panel experts are given an opportunity to discuss individual views on the appropriateness of the intervention in each clinical scenario. At the end of the discussion each panelist can reconsider their original rating and re-rate the clinical scenario. The results are then summarized as descriptive statistics. Disagreement is when one-third or more panelists rate scenario in lowest 3 points and one-third or more rate same scenario in highest 3 points of appropriateness scale. In the absence of disagreement median rating in lower 3 points is “inappropriate”, median rating in upper 3 points is “appropriate”, and rating in the 4–6 range or those with disagreement is “uncertain.” These ratings can finally be used to determine whether an intervention has been used inappropriately retrospectively or determine whether an intervention should be performed prospectively. Based on the number of participants included, there can be some variation in the definitions of agreement.

Based on the ratings of the expert panel an indication in each clinical scenario will be considered, appropriate, uncertain, or inappropriate, however it lacks clear rating of the evidence or a ranking of the scientific robustness of the recommendations.

While RAM was initially used for assessing appropriateness of a procedure, it has been applied to the development of practice guidelines and classification criteria. Once indications have been ranked by appropriateness using a Likert scale as described above, summary statements reflecting the output from the consensus exercise are used to make a set of classification criteria or practice guidelines (50). In making summary statements, both the perceived appropriateness of a certain intervention or guideline as well as the agreement among participants is critical.

RAND-UCLA Appropriateness Method

Advantages

Disadvantages

Synthesis of published literature prior to consensus techniques incorporated Allows for both confidential ratings as well as group discussion Multi-disciplinary panel encourage consensus from a wider group Reproducibility of RAM ranges from moderate to excellent as determined by different panelists for “appropriate” and “inappropriate” care (53). Acceptable predictive validity for a recommendation supported by RCTs(54).

Misclassification is expected (49) Takes great deal of time from gathering of the evidence to multiple rounds of consensus. Face-to-face which can add cost/time delay and lead to highly opinionated individuals in the field dominating the discussion Requires third party (core panel) to construct clinical indications for an intervention and analyze/interpret the results from the expert panel meeting 9-point Likert scale can be cumbersome Requires voting on multiple case scenarios (sometimes > 1,000)

Examples in Rheumatology

This methodology has been used for development of quality indicators in rheumatology (51, 52). Recommendations have been developed using this methodology including recommendations for clinical agent evaluation for treatment in osteoporosis which was endorsed by the American Society for Bone and Mineral Research (ASBMR), the International Society for Clinical Densitometry (ISCD), and the National Osteoporosis Foundation (NOF) (53). Recently, quality indicators for systemic lupus erythematosus (54) have been published as well as recommendations for evaluating and treating rheumatoid arthritis (10, 55).

Recommendations for use

Any project that needs a formal method for combining scientific evidence from the literature and expert consensus, might consider RAM. Specifically, it can be used in the:

1. Developing clinical practice guidelines where evidence is not sufficient.

2. Developing quality of care indicators.

3. Determining appropriate use criteria for a medical intervention.

4. Determining overuse or underuse of a procedure.

This rigorous method can be used when resources and time are available and can be used in any circumstance where thorough combination of scientific evidence and expert opinion is required.

Consensus Development Conference (CDC)

In 1997, the National Institute of Health (NIH) in the United States introduced the consensus development conference (CDC) (8). The CDC brings together selected experts in the field and concerned individuals (from expert to public) to reach general agreement about the safety, efficacy, and appropriateness of using various medical procedures, drugs, and devices. The Office for Medical Applications of Research (OMAR) of the NIH is charged with responsibility of this conference (56). OMAR's primary goal is to help bring the results of biomedical research into direct use in the practice of medicine. To this end, it acts to coordinate consensus development activities at the NIH for evaluating current evidence and promulgating consensus of opinions on a particular topic. Each conference is jointly sponsored and administered by one or more Institutes or Centers (ICs) of NIH and by the OMAR. They have run more than 100 conferences on a variety of topics (57). The development of consensus conferences is a hybrid of methods used by judicial decision-making, scientific conferences and the town hall meeting (58).

Process

A topic for the CDC, usually of public health importance, is decided by agreement between the sponsoring IC and OMAR. A set of predetermined questions on which a consensus is warranted are selected from the topic by experts appointed by sponsoring IC and OMAR, which defines the scope of the conference. A different selected group of experts (around 10 experts) is invited and brought together to form an expert panel or decision making group for the conference. The panel is an independent, broad-based, non-Department of Health and Human Services (DHHS), non-advocacy group with appropriate expertise. The panel members are highly regarded in their own fields but are not closely aligned with the subject. The evidence is then presented to the conference by various experts appointed by sponsoring IC and OMAR, who are not a member of the decision making group. Usually, the Agency for Healthcare Research and Quality (AHRQ) provides a systematic review of literature on the conference topic through one of its Evidence-Based Practice Centers. The group or expert panel hears the scientific data presented by invited experts as well as the comments from the general public in a public (open) session followed by discussion. The format allows the participation of the audience (members of the public) in an open meeting, who can ask questions to the group. The panel then meets in a private (executive) session to further deliberate on the evidence and discussion to reach a consensus. The chairperson is the moderator responsible for guiding and controlling the proceedings of both the open part of the conference and the private group discussion as well as helping reaching consensus on topic/question where 2 or more experts differ in their opinion. The panel weighs the information and reaches a consensus statement that addresses a set of predetermined questions. The consensus statement draft is then presented in a plenary session and is subject to review and comment by conference attendees. Following the discussion, the panel may then modify the statement if appropriate in the final executive session. The final consensus statement is then released and disseminated widely to achieve maximum impact on health care practice and medical research. This is an independent report and is not considered a policy of NIH or the Federal Government. A major contribution of the NIH consensus panels has been to describe current levels of agreement on important topics like coronary artery bypass surgery, intraocular lens implantation, cesarean section, Reye' syndrome, and the treatment of breast cancer.

Consensus Development Conference

Advantages	Disadvantages
Mix of practicing physicians, researchers, consumers, and others to come together and jointly evaluate an existing technology Wide circulation through both lay and medical media Unbiased panel	Interaction is not structured The aggregation methodology used is implicit- a formal feedback system is lacking CDC has not been used for making new criteria sets

Examples in Rheumatology

Use of CDC in the field of rheumatology has been limited to systemic review of diagnosis and therapeutic options on the topics like osteoporosis, total knee and hip replacement as well as recommendations for IVIG therapy (68–71). There have been no criteria set developed by NIH CDC meeting in any field of medicine including rheumatology.

Recommendations for use

1. CDC conferences are particularly useful for providing guidance when a controversy exists over preventive, therapeutic, or diagnostic options for public policy

2. When the issue is of high degree of public interest or has impact on health care cost.

It is intended as a statement that reflects the views of expert panel that have examined and discussed the scientific data, rather than a legal document or a practice guideline. The statement may reflect uncertainties, options, or minority viewpoints.

Use in Rheumatology

Use of CDC in the field of rheumatology has been limited to systemic review of diagnosis and therapeutic options on the topics like osteoporosis, total knee and hip replacement as well as recommendations for IVIG therapy (59–62). There have been no criteria set developed by NIH CDC meeting in any field of medicine including rheumatology.

Grading of Recommendations Assessment, Development and Evaluation (GRADE)

This review also discusses GRADE as it has been adopted by scientific world for the development of recommendations. GRADE is not a consensus methodology per se but uses consensus methods discussed above (especially Delphi and NGT) to assess quality and strength of a recommendation. GRADE methodology provides a systematic and transparent approach to rate the quality of evidence and grade the strength of recommendations for patient important outcomes. GRADE was developed by experts with a goal to answer some drawbacks of previous methods, which include the lack of separation between quality of evidence and strength of recommendation and the lack of explicit acknowledgment of values and preferences (63). Several organizations and guideline developers, including the World Health Organization, the American College of Chest Physicians the American Endocrine Society, and UpToDate, have adopted the GRADE system. GRADE Quality of evidence is graded into 1 of the 4 levels— high, moderate, low, and very low. RCTs usually get a higher grading compared to observational studies. The system allows the quality of evidence derived from observational data to be upgraded from low to moderate or high categories and the quality of evidence coming from randomized trials to be downgraded depending on the design and execution of such studies. The strength of recommendation is divided into strong (benefits far outweigh risks or that risks far outweigh benefits and virtually all patients will make same decisions) and weak (where as tradeoffs between risks and benefits is less clear and patient’s values and preferences are incorporated before making a treatment plan). Factors influencing strength of recommendations may include quality of evidence, balance between desirable and undesirable effects, individual’s values and preferences for a treatment, and resource utilization (63, 64).

Examples in Rheumatology

American College of Rheumatology guidelines in osteoarthritis (unpublished). Other examples in the literature include use in reaching decisions on practice guidelines (65) and grading recommendations for the use of thrombolytic agents and prevention of coronary artery disease (64, 66).

Recommendations for use

1. To develop clinical guidelines.

Limitations of Review

Although we present a comprehensive review of formal consensus methods, we didn’t conduct a systematic review. In our comprehensive review, the majority of the articles on formal consensus methodologies are implementations in diagnostic criteria, guidelines, and response indices development.

Conclusion

Formal consensus methodologies have various applications in the medical literature from guideline development to development of criteria sets. These methods are being increasingly used in rheumatology. Each methodology has it unique attributes and utilization of a particular methodology or combination of methodologies depends on: 1. clinical question, 2: audience, and 3: available resources.

When closely tied to evidence-based literature, these summarized statements can provide physicians with guidance in classifying and treating disease. Dangers with group consensus occur when there is not rigor in the methods used and when the ensuing recommendations are non-specific (67). Also, consensus methodologies provide areas/ strength of agreement (and disagreement) and should not replace evidence-based literature.