Consensus development for healthcare professionals

Abstract

Consensus development sprang from a desire to synthesize clinician and expert opinions on clinical practice and research agendas in the 1950s. And since the American Institute of Medicine formally defined “guidelines” in 1990, there has been a proliferation of clinical practice guidelines (CPG) both formally and informally. This modern decision making tool used by both physicians and patients, requires extensive planning to meet the challenges of consensus development while reaping its rewards. Consensus allows for a group approach with multiple experts sharing ideas to form consensus on topics ranging from appropriateness of procedures to research agenda development. Disagreements can shed light on areas of controversy and launch further discussions. It has five main components: three inputs (defining the task, participant identification and recruitment, and information synthesis), the approach (consensus development by explicit or implicit means), and the output (dissemination of results). Each aspect requires extensive planning a priori as they influence the entire process, from how information will be interpreted, the interaction of participants, the resulting judgment, to whether there will be uptake of results. Implicit approaches utilize qualitative methods and/or a simple voting structure of majority wins, and are used in informal consensus development methods and consensus development conferences. Explicit approaches aggregate results or judgments using explicit rules set a priori with definitions of “agreement” or consensus. Because the implicit process can be more opaque, unforeseen challenges can emerge such as the undue influence of a minority. And yet, the logistics of explicit approaches may be more time consuming and not appropriate when speed is a priority. In determining which method to use, it is important to understand the pros and cons of the different approaches and how it will affect the overall input, approach, and outcome.

INTRODUCTION

Clinicians make multiple clinical decisions every day, and ideally, these decisions would be based on rigorous evidenced-based guidelines. Much of health care practice however, has relied on expert opinion or been driven by clinician experience. Compared to other industries, consensus development in health care practice and research is a relatively young field with a growing body of data [1]. In the 1950s, in an attempt to synthesize clinician and ‘expert’ opinion for clinical practice guidelines and research agendas, the RAND Corporation (a nonprofit, non-partisan global public policy think tank) developed the first approach to consensus guideline development –the Delphi method, which uses iterative surveys without structured interaction to aggregate judgments. Studies using the Delphi method in healthcare and research have included topics on: research priorities for trauma nursing [2], clinical nursing research priorities [3], agreement on classification of electrocardiograms [4], and agreeing terminology for substance abuse [5]. However, this method has been phased out in favor of methods that include the benefits of structured interaction, such as Nominal Group Technique (NGT) and modified Delphi method.

The modified Delphi method, also known as the RAND/UCLA method, popularized by Brook et al of RAND [6, 7], convened three expert panels (made up of a multi-disciplinary set of physicians who were leaders in American medicine). After reviewing synthesized literature, the panels rated the appropriateness of six medical and surgical procedures for a set list of indications in cardiology (coronary angiography and coronary artery bypass surgery), cerebrovascular (carotid endarterectomy), and gastroenterology (colonoscopy, endoscopy and cholecystectomy), prior to meeting and then during a meeting. The study showed it was possible to produce appropriateness ratings of procedures for a large number of indications at a lower cost and logistically less intensive process than a randomized control trial. The predictive validity of such “softer science” methodology has come in the form of multiple randomized control trials (RCT). Shekelle et al 1998 analyzed the results of 7 subsequent RCTs on carotid endarterectomy (CEA) to assess the predictive validity of the appropriateness criteria from the RAND/UCLA study. The RCTs confirmed 44 of the indications for CEA (covering 30% of the operations performed in 1981) and refuted no ratings [8]. Thus, while not generating new information, there is added-value to consensus development until there is an RCT available.

Since the American Institute of Medicine’s formal definition of “guidelines” in 1990, clinical practice guidelines (CPG) through consensus have become the modern decision-making tool for patient and clinicians for specific clinical scenarios [9]. Both fortunately and unfortunately, formal and informal guidelines have proliferated making it difficult to users to judge the quality or appropriateness of the guidelines for a specific population [10–12]. As a result, there has been a growth in guideline clearing houses to systematically identify, review and organize guidelines. From national guideline clearing houses (such as the United States (US) Agency for Healthcare Research and Quality (AHRQ) Guideline Clearing House, United Kingdom (UK) National Institutes for Health and Clinical Excellence), condition level clearing houses, to professional societies have been developed [1]. Since then the National Guideline Clearing House adopted the American Institute of Medicine primer on guideline development, Clinical Practice Guidelines We Can Trust [13].

This group approach to guidelines or research agenda development allows for an expanded range of knowledge and experience by multiple individuals, stimulates a deeper discussion with the inclusion of expert recommendations, challenges existing ideas, and increases the legitimacy of the output. Furthermore, the results are more likely to have both concurrent validity and internal logic, otherwise “closer to the truth.”

However, there are also inherent challenges to the group approach as well. For example, the methods of a group interaction can influence how the group arrives at the truth and the resulting output [1]. Additionally, selection of the “right” participants may include research scientists, clinical decision-makers and patients who may all have different agendas. Thus, management of the interaction must be considered a priori in order to prevent dominance by a minority (e.g. high status members, individuals with strong opinions or personalities) while simultaneously ensuring contribution of introverted experts. Furthermore, if a group is too cohesive or there is undue influence of a minority or majority occur, the output is at risk of group think, group polarization, or minority influence [14]. And of course with larger groups, the logistics of organizing more people is a factor for both cost and resources.

GOAL

The objective of this article is to review the common approaches to consensus development and methodology, as well as the benefits and negatives of consensus. Specifically, this article will review examples in the health care literature, for which there is a slow growing body of literature, and on the development of quality indicators and future research agendas. Schunemann et al has published exhaustively on guideline development and systematic reviews, and refer the interested reader to these papers for details [15, 16]. The development of clinical practice guidelines have also been extensively covered by Shekelle et al [17–19], and their methodology will be briefly reviewed here.

OVERVIEW

The structure of consensus development can be categorized into five components: three inputs, the approach (consensus development), and the output [18]. In the input, tasks must be defined, participants are identified and recruited, and information synthesized (literature review and synthesis). The aggregation of judgments, the approach, can take on two methods: implicit and explicit. The implicit approach utilizes qualitative methods and/or a simple structure of majority vote, and is used in informal consensus development methods and consensus development conferences. The explicit aggregation of judgments involve more complex statistical methods (with or without weighting of judgment, means of individual judgments, mathematical integrations) to form consensus, and is exercised in the Delphi, NGT, and modified Delphi method. The output depends on the audience, but should use the active language to provide recommendations.

I. Defining the Task

Every group is tasked to address one or more questions proposed by the organizers. In some consensus conferences, these may be just a few questions, whereas with the Delphi method and NGT, they can include numerous questions that narrow after an initial survey. In clinical practice guidelines (CPG), tasks are defined by a desired clinical behavior and specific outcomes of interest (intermediate, surrogate, and health outcomes) are addressed a priori to making recommendation, such as identifying “electrocardiogram changes as a surrogate for cardiac ischemia” [19]. The primary task should have an overarching outcome of interest, and may include hundreds of subtasks for specific clinical scenarios. These subtasks are rated for appropriateness of diagnostic or interventional actions, similar to the modified Delphi method.

This approach can be extended to research agendas as well where broad research questions are ranked and prioritized for impact, with subtasks for each question. This list is then refined with further input, or feedback from sponsoring groups, until a research agenda is formed with tens of agenda items. These tasks can be defined by sponsoring organizations such as the US National Institutes of Health (NIH), US Preventative Service and Task Force (USPSTF), or UK National Health Services, or by professional societies such as American College of Emergency Medicine or American Thoracic Society, or smaller interest groups such as the Pediatric Emergency Care Applied Research Network [18, 17, 19].

In an iterative approach, defining tasks can be first characterized by a literature review that includes synthesis and methodological grading. Subsequently, a conceptual model is developed to structure the task and focus participants [1]. The initial tasks are then defined based on participants’ perspectives, sponsor requirements, and gaps in the literature. An advisory group, non-voting stakeholders not part of the guideline group, can provide feedback to further refine the tasks.

II. Identifying and Recruiting Participants [1]

The participants of a consensus development group play a major role in the recommendations or output of their interaction. Thus, the composition must be determined a priori where the expertise can range from clinicians (who implement the guidelines and are decision-makers), research scientists (investigating the questions methodologically) to patients (who have direct experience with the disease) [1]. For example, in one RAND/UCLA study [6, 7], experts rated 6 procedures in 3 categories. For cardiac procedures, experts included cardiologists, internists, family physicians, cardiothoracic surgeons, and a radiologist. For gastroenterology procedures, experts included a family physician, internists, gastroenterologists, general surgeons, and a radiologist. For cerebrovascular procedures, a family physician, internist, neurologists, vascular surgeons, a neurosurgeon, and a neuroradiologist were included on the panel. This heterogeneity can provide unique perspectives, alternative approaches in problem-solving, increased idea-sharing and buy-in from other fields [1].

However, if subgroups develop into cohesive groups with polarized views, there is a potential for conflict as well [1]. Constructive conflict resolution may be necessary when there is considerable controversy. And yet, if there are differing opinions among experts, participants are likely drift to the mean. Such diversity is also useful for the development of research agendas or exploration of areas of uncertainty.

A homogenous group in contrast, is more likely to reach agreement more quickly than a heterogeneous group, and will tend to have consistent results when compared across similarly composed groups. This increased validity is useful for finding common ground quickly but may lead to particular judgments due to polarized homogenous views.

Who

Most consensus development groups include experts in the field of interest who have credibility in the target audience as it may improve the validity [1]. Identification of such individuals may be based upon professional society recommendations, number of publications, administrative leadership, reputation or peer consensus. Individual with higher status however, are likely to dominate an interaction with a greater influence on the group. And while their input is important, it is equally important to include the input of other members. Therefore, methods to mitigate the unbalanced input of a few participants should be undergone (e.g. confidential voting) [1].

Another set of stakeholders, increasingly included in expert panels, are the patients. With the rise of the US Patient-Centered Outcomes Research Institute (PCORI), the focus of patient involvement is changing. PCORI was created as part of the Affordable Care Act (ACA) in the US, President Obama’s universal health insurance initiative for Americans without health insurance, out of a need to address patient- and clinician-driven questions in real-world settings. It is financed through the US Treasury and a private health insurance fee. As an independent body from the government, it is solely a research funding institution that aims to generate information. Grant applications undergo a peer-review process that includes 3 patients or stakeholder reviewers. PCORI believes that patients and family members with direct experiences with a disease are able to contribute in a meaningful way as they are able address issues important to them or ask more patient-pertinent questions. All projects funded by PCORI have an explicit requirement to include all stakeholders (physicians, nurses, caregivers, family members, pharmaceutical industry, etc…) with a “stakeholder engagement plan” where there is meaningful involvement [20]. Patients are also on PCORI’s grant review committees, thus, they have a prominent role in multiple aspects of research.

Optimal number

When there are more judges, the reliability of the composite judgment increases and is presumed to be more accurate [1]. Hogarth et al 1978 found that group validity was closer to the “truth” with more than five participants [21], thus most literature indicate at least six participants is more reliable. While the reliability will improve with greater numbers, there are diminishing returns with numbers greater than 12 as it is may be logistically more difficult to coordinate [1]. However, current technology can potentially incorporate many participants with the use of web-based surveys for different aspects of consensus development [22].

Conflict of interest

An important aspect gaining attention in healthcare, research and CPG, is the disclosure of potential conflicts of interest [23]. Many journals and conferences now require authors and presenters to disclose their conflicts of interest which may include testimony, stock holding, honoraria, in kind gifts, pain expert testimony, or acting board members. This conflict represents bias that could influence not only the question to be addressed but also the proceedings and final recommendations. Discovering such undisclosed conflicts can adversely affect the reputation of those setting forth the guidelines, such as with the Surviving Sepsis Campaign [23].

Eichacker et al 2006 startling article uncovered the multiple conflicts of interest in this campaign [23]. In this campaign, Eli Lilly’s Xigris (recombinant human activated protein C, or rhAPC, also known as drotrecogin alfa-activated) was included into performance bundles for sepsis treatment despite controversial evidence from a single phase 3 randomized, control trial—the Recombinant Activated Human Protein C Worldwide Evaluation in Severe Sepsis (PROWESS study) [24]. The US Food and Drug Administration (FDA) approved the drug contingent on prospective validation on subgroups due to a high risk of death and hemorrhage associated with the drug. As a result, sales of the drug declined and Eli Lily hired a public relations firm to improve sales. They launched a three pronged approach to improve sales: 1-target physicians and medical trade media, 2-because it was such an expensive drug, they created an environment of “rationing” such that physicians felt they were deciding who would survive if given the medication, and 3-establish the Surviving Sepsis Campaign to increase awareness about sepsis and develop guidelines on treatment. Eleven societies endorsed the guidelines, and held workshops on sepsis at their meetings supported by Lily. Lily also sponsored a popular quarterly periodical, Advances in Sepsis, that furthered their campaign and calls for new prospective studies of rhAPC.

The company supported multiple individuals in different societies and had a number of articles published where authors were board members of the company. Despite this undue sponsor influence, the guidelines were endorsed by multiple societies. The Infectious Disease Society of America was the only society that found methodological flaws in the guidelines and rejected the guidelines; however, it was not widely publicized. Two further studies, ADDRESS in 2005 [25] and RESOLVE in 2007 [26] warned of issues with rhAPC, but the campaign ignored these studies until the PROWESS-SHOCK study in 2011 [27] when Lily withdrew it from the market. Preventing the creation of conflicted guidelines requires declaration of potential participant conflicts of interest prior to their involvement in guideline development.

III. Information

Information, or evidence, given to participants will strongly influence their judgments [28]. Without information, participants rely on past experiences to form them [29]. As a result, organizers must determine the quantity of information to provide, the selection of evidence and how to present it as it will affect how participants assimilate, interpret, and use the information. Too much information will overwhelm participants, but too little will not provide enough data. By providing a literature review and/or library of articles and abstracts, participants will have equal access to all available data and create a common starting point for discussion [1].

Systematic reviews are a crucial component of many CPGs as they may summarize the current literature for the proposed CPG question. The American Institute of Medicine developed standards in 2011 for their development which is organized into four areas: (1) initiating a systematic review, (2) finding and assessing individual studies, (3) synthesizing the body of evidence, and (4) reporting of systematic review. However, how panelists evaluate the literature, including systematic reviews, may impact the outcome of the CPG. Thus, methods assessing quality of a systematic review is essential to the CPG process. While there are multiple tools available, two popular methods include AMSTAR (Assessment of Multiple Systematic Reviews) [30, 31], and GRADE[32, 16, 33–52] (Grade of Recommendation, Assessment, Development and Evaluation), which aims to develop a common, transparent sensible system for grading the quality of evidence and the strength of evidence. This method has been taken up by multiple organizations including the World Health Organization, the Cochrane Collaboration, the National Institute of Clinical Excellence, UpToDate, and many other organizations. GRADE Profiler Software is freely available for downloading: http://tech.cochrane.org/gradepro.

While the systematic reviews, original articles and abstracts are less likely to be misinterpreted, presenting synthesized data in the forms of tables are easier for participants to assimilate and use the data [1]. Multiple studies have shown that how data and outcomes are presented will influence judgments [53–57]. In one Swiss study, physicians were asked to rate the effectiveness of drug treatment for hyperlipidemia. One questionnaire presented the data as absolute risk reduction for non-fatal and fatal myocardial infarction combined, fatal myocardial infarction alone, and total mortality. In a second questionnaire, the same data was presented as relative risk reduction. The effectiveness ratings for relative risk reductions were higher than absolute risk reduction. Thus, data syntheses and presentation must be carefully planned a priori such that participants are less likely to misinterpret the data.

Prior experience is also a form of information that is can influence how new information is interpreted. Experts are more likely to rely on experience but also may be better apt in reviewing literature for relevant articles in their area of expertise [1]. The involvement of methodologists during a literature review will likely improve the quality of the review. A methodologist is more likely to be better weight relevant, methodologically sound and appropriate literature during a synthesis. Subsequently, the judgments based on the reviews toward a more appropriate judgment [1]. Ultimately, judgments are influenced not only by the selection of literature reviewed, but also the impact of values and preferences for various outcomes by panelist.

Another aspect of information is feedback after voting. Feedback can be presented in different forms, but often includes the scores of individual participants, all participants anonymized and the group scores with means and medians. Such information is critical prior to another round of voting in the Delphi, modified Delphi and NGT methods as forms of structured communication. This iterative process is likely to decrease the dispersion of the voting and improve convergence and accuracy of a judgment, but not necessarily validity [1].

IV. Consensus Approaches

The aggregation of judgments can take on two approaches: implicit and explicit. The implicit approach utilizes qualitative methods and/or a simple voting structure of majority wins, and is used in informal consensus development methods and consensus development conferences. In contrast, the explicit aggregation of judgments involve more complex statistical methods (with or without weighting of judgment, means of individual judgments, mathematical integrations) to form consensus, and is exercised in the Delphi, NGT, and modified Delphi.

Implicit approaches often begin with a literature review. Ideally, participants are given a selected methodology for grading the evidence, on quality, quantity and consistency across the literature, with consideration for publication bias [19]. This information is used in the group’s discussion and then views are summarized by the facilitators. While easy to administer, the process may be opaque and dissent hidden. And yet, it may be the only available approach when certain questions do not lend themselves to a multiple item format for NGT, such as: What are the top research priorities for a certain clinical specialty? Most professional societies releasing CPG utilize this approach.

The explicit approach is similarly first characterized with a literature review that includes simple narrative summaries, evidence tables, meta-analyses, and modeling [19]. Using this information and their own perspectives, participants undergo several rounds of anonymized voting. Results are then aggregated using explicit rules set a priori. Furthermore, there are explicit definitions of “agreement” or “consensus,” such as a certain percentage of concordance and lack of discordance equates agreement. Due to the multiple rounds of voting and explicit rules, it is more difficult to administer than the implicit approach.

The Delphi method developed in the 1950s by RAND attempted to reap the benefits of low-cost information (as members may have a wide geographic distribution) but limit the detriments of interaction [1]. Questionnaires are sent to individual participants and responses are summarized anonymously (also known as the forecast) and then returned back to members with the inclusion of their initial response. Members are encouraged to revise their response in light of the new information received. This iterative process can be continued multiple times. Votes can be statistically weighted for member expertise, and then responses are statistically aggregated. Critics however, remarked that the lack of interaction leads to less discourse about disagreements and does not provide an opportunity for explanations of perspectives. As a result, it has fallen out of favor.

In contrast to the Delphi method, the NGT utilizes structured interaction where members privately record their ideas, and then in a round robin style voice their idea until all unique ideas are recorded. The group then discusses each idea listed, and then members privately rank the ideas based on the new information. Individual rankings are statistically aggregated to derive a group judgment. Because there are two phases, the first phase of idea generation increases participation by more inhibited members and increases the number of ideas as all members must give an idea. Thus, this is an excellent method for idea generation but may be difficult to reach a valid consensus if there are multiple heterogeneous ideas without a particular direction and lack of iterative process to reach consensus.

Because of the criticism of the original Delphi method, a modified Delphi method was developed—also known as the Rand form of the NGT, an amalgamation of both the original Delphi and NGT [6]. A group of 9 experts are chosen and must agree upon a set of decision-making scenarios. Each expert ranks a scenario 1 through 9, where 1 is extremely inappropriate (risk greatly exceed benefit) and 9 is extremely appropriate (benefits greatly exceed risk). These votes are aggregated and discussed as a group. In phase 2, each expert again casts their votes privately and then appropriateness scores are calculated based on the median. Agreements and disagreements are determined based on a statistical definition using binomial distribution. This approach allows for both structured interaction and explicit aggregation of votes.

Strategies for improving approach

Implicit

Based on prior experience of the author (BCS), challenges can arise when members dominate the discussion or if members do not commit to form consensus. Potential solutions include ground rules for discussion (round robin, parliamentary procedure, etc…) and a priori commitment to obtain consensus by all members.

Explicit

Strong disagreements can also be challenging, and determining a process to address it a priori can help alleviate downstream tension and conflict. In modified Delphi, disagreement is defined by four definitions where they note the degree of disagreement within a 3 point range and the entire 9 point range for both all participant ratings and then the ratings that remained after discarding the extreme high and low rating. While disagreement can be conflict provoking, useful information can be obtained with structured disagreement as it can uncover controversial topics.

V. Output

The dissemination of results has several approaches and must be considered a priori given the context of the audience. There may be a primary audience and a secondary audience. Not only is the scientific community impacted by new guidelines, but also the lay public with potential policy ramifications. For example, in 2009 the U.S. Preventative Services Task Force convened an independent group of experts (implicit approach) that deemed yearly mammograms were no longer necessary for all women younger than 50 (unless they had a family history of breast cancer or other risk factors) [58]. Despite using the most up-to-date literature, there was an immediate backlash by patient advocacy groups and professional societies [59]. Women saw it as a threat to their lives and their daughters lives and triggered an enormous amount of media coverage. Even the White House became involved as it affected President Obama’s new health insurance policy. As a result, the USPTF modified their language such that decision to start regular, biennial mammograms before age 50 be an individual one and include the consideration of patient values [58]. The negative press however, increased the skepticism of women regarding their physicians’ perspectives on mammography.

Such negative press and distrust from the public ought to be prevented. Methods to avoid such an incident may include peer review or consultation with the public to critically review drafts and provide feedback prior to final publication. This group however does not have voting capabilities but allows the opportunity for increased validity, alternative viewpoints, and “buy-in” by a heterogeneous group. Explicit rules, and/or transparent methods, for inclusion or ignoring criticism must be created in order to justifiably invoke and ignore feedback [18].

Guidelines and agendas are often published in scientific journals, but strategies to support guideline uptake may include development of educational resources, using multiple formats and channels for guideline dissemination based on preferences of the target group, use of behaviorally specific language in the guideline, pre-emptive identification of potential barriers of recommendations, and a priori generation of solutions to address them by the guideline development group, use of data collection tools, and identification of resource implications of recommendations [18, 60].

EXAMPLES OF CONSENSUS APPROACHES IN THE HEALTH CARE LITERATURE

In the emergency medicine literature, consensus development on quality measures and guidelines has primarily been approached through the modified Delphi method. This approach is best used where a pre-identified set of items are easily rated, and structured interaction is conducive to further discussion, followed by feedback and re-voting to bring about consensus on a more narrowed scope. In research agenda setting, both structured and unstructured approaches have been used. If speed is priority, the unstructured approach is best, however, a structured approach will likely have more transparency and increased consensus.

NGT/Modified Delphi

1. Penciner et al 2011 conducted a study to establish consensus on emergency medicine clerkship competencies in Canada using modified Delphi process [61]. They convened an expert panel of thirty persons from nine different medical schools. An initial list of 152 competencies were rated in the round 1, and reduced to 62 after a second round of the Delphi process.

2. In Lindsay et al 2002 on ED quality indicators were developed using the modified-Delphi approach [62]. After a literature review to identify conditions frequently treated in the ED and the outcomes associated with the condition by the authors, an expert panel was convened to rate specific clinical condition-outcome pairs where quality of care could improve the outcome for the condition through a questionnaire. As a result, another literature review was performed to identify measurable clinical indicators of performance for these pairs. Participants then rated these indicators for appropriateness using a Likert-type scale (1-“strongly disagree” to 9 “strongly agree”) in another questionnaire. In a structured interaction, panel participants were given both individual and summarized results and then allowed to discuss each indicator in turn similar to the NGT technique. Panelists then individually rated the indicators in light of the new information. Core indicators were selected based on strong agreement between panelists.

3. Solberg et al 2003 study on ED crowding measures [63], used both the modified-Delphi and NGT techniques to develop ED quality measures using structured interaction and web-based anonymous voting. A core set of investigators and 74 expert participants developed 6 rating categories for measures, and elicited ED quality measures outlined by their conceptual model from all experts in an anonymous fashion. Duplicate measures and those not meeting criteria were eliminated. In a structured interaction, core investigators further refined the measures (by combining, rewording and eliminating those of little value). These measures were then rated by anonymized expert participants via a website using magnitude estimation technique. Summarized ratings and comments underwent another review with core investigators during another meeting and added 8 more measures which were then rated on a website by expert participants as well, and resulted in a final set of ED crowding measures

Internet-based survey

1. In Sun et al 2012 study on ED syncope risk-stratification reporting guidelines [22], guidelines were developed using an expert panel modified Delphi process. Using internet-based surveys in an iterative approach, experts anonymously rated tasks in two rounds. After the first round, both summarized and anonymized individual ratings were given back to participants for review. Further, structured interaction was encouraged through moderated conference calls and e-mail forums. And then a second round of internet-based surveys took place privately. Those items achieving greater than 80% consensus were included in the final guidelines.

Unstructured research agenda setting

In 2010, research consensus conferences were held with US National Institutes of Health (NIH) Task Force members and academicians, to prioritize opportunities for facilitating and conducting emergency care research to advise the new Office in Emergency Research at NIH [64-66]. With the multi-disciplinary reach of Emergency Medicine both clinically and within research, it was critical for NIH to have clear research agendas established to justify targeted research funding. These round table discussions advised the new NIH Office of Emergency Care on emergency trauma, medical-surgical, and neurologic and psychiatric emergency research.

1. In emergency trauma and the medical-surgical emergency roundtables, experts within the field were invited to list challenges and research priorities in a pre-conference survey. This list was revised during and after the conference until consensus was reached [64, 66].

2. In the neurologic and psychiatric emergencies roundtable, a multidisciplinary planning group developed an initial agenda, selected participants and presenters, and based on the state of the science, burden of disease, and request for information responses, narrowed the final research agenda by consensus voting during a roundtable conference [65].

Structured research agenda setting

1. In the Italian syncope project, an international multi-disciplinary panel of syncope experts identified research priorities in a preconference survey on syncope, developed a conceptual model of ED decision-making for syncope, and then refined during and after the conference through an iterative approach until priorities for research were published [67].

CONCLUSION

Consensus development can be used for both clinical practice guidelines and research agendas. ‘Expert Consensus’ is important when no evidence exists for most clinical/research scenarios. In determining which approach to use, it is important to understand the pros and cons of the different approaches and how it will affect the overall input, process, and outcome. Because the implicit process can be more opaque, unforeseen challenges can emerge such as the undue influence of a minority. Also, lists developed without a limitation to the number of items or prioritization can make these lists potentially useless. Careful planning, however, can mitigate both of these situation using structured interaction as well as clear aims and priorities set prior to meetings. Its primary strength compared to explicit approaches is the rapid assembly of “experts” and dissemination through journals. While explicit approaches may result in more valid and transparent results, it logistically requires significantly more effort and resources than implicit approaches.

Table 1.

Summary of consensus development methods.

	Implicit Methods	Explicit Methods
Approaches	Informal consensus development, consensus development conferences, unstructured research agenda setting	Delphi, NGT, modified Delphi (or RAND/UCLA), structured research agenda setting
General methodology	Qualitative approach and/or simple structure of majority vote, usually of experts	Structured interaction (for NGT and modified Delphi) with explicit rules for aggregation of judgments with complex statistical methods
Pros	Easy to administer, rapid assembly of experts and dissemination	Results are likely more valid, transparency in methods
Cons	Opaque process, hidden dissent, unclear structure of interaction, potential greater influence or dominance of interaction due to participant status, long lists without prioritization	Logistically more difficult to administer, time consuming
Examples	Most clinical practice guidelines released by professional societies; roundtable US NIH discussions to set research priorities in EM	Consensus on ED quality indicators, ED crowding measures, research priorities in syncope
When to use	Research agenda setting where there are multiple research priorities, speed is a priority, logistically infeasible to do explicit methods, lack evidence for most clinical/research scenarios	When not limited by time, logistics, and evidence

Abbreviations: NGT, nominal group technique; ED, emergency department; EM, emergency medicine; US NIH, United States National Institutes of Health