Educator's blueprint: A primer on consensus methods in medical education research

Abstract

Consensus methods such as the Delphi and nominal group techniques are increasingly utilized within medical education research. This educator's blueprint paper provides practical strategies regarding five key steps for ensuring best practices when using consensus methods. These strategies include deciding which consensus method is best, developing the initial questionnaire, identifying the participants, determining the number of rounds and consensus threshold, and describing and justifying any modifications. These strategies will help guide education researchers on their next study using consensus methods.

INTRODUCTION

Decisions surrounding best practices in medical education should be based on empiric evidence; however, this evidence is frequently lacking. As a result, decisions are made by relying on the judgment of experts and if done informally may lack credibility. Consensus methods are formal structured group communication processes that can be used to synthesize expert judgment in a rigorous fashion. ¹ They can be used to assess the extent of agreement (consensus measurement) and/or to resolve disagreement (consensus development) depending on the purpose of the exercise. ¹ Consensus methods are employed with the intent to overcome disadvantages found in group decision making such as domination by one or two strong individuals or a select group of strongly represented interests. ¹

Consensus methods are commonly used in medical education research. When we conducted a search of consensus‐based methods (specifically, nominal group and Delphi techniques) in medical education within PubMed, we identified a threefold increase in the number of studies published using these techniques compared with 10 years prior. ² This rise in usage is not surprising, given the potential applications and broader recognition of these approaches. ³ These tools can help understand the degree of agreement and develop consensus across an array of topics, including curriculum development, assessment tool development, and defining competencies or entrustable professional activities. ² , ⁴ , ⁵ , ⁶ , ⁷

However, the literature has demonstrated substantial variability in the conduct and reporting of consensus‐based research. ² , ⁸ , ⁹ Therefore, there is a critical need to better understand the best practices in consensus methods to inform local usage, research reporting, and interpretation of the literature among education scholars. This Educator's Blueprint paper will present five key steps to help ensure high‐quality consensus methods in education research.

PRACTICAL STRATEGIES FOR FIVE KEY STEPS

1. Decide which consensus technique is best

There are a variety of methods for reaching consensus (e.g., nominal group technique, Delphi, RAND, informal consensus). ² Which method you select depends on several factors. While several consensus‐based methodologies exist, we will focus on nominal group technique and Delphi approach in this paper, as these reflect the overwhelming majority of those used in medical education. ²

Both nominal group and Delphi methods use multiple rounds of questionnaires or surveys to ultimately measure consensus using the opinions and feedback of experts in a given field. ¹⁰ The Delphi method involves polling experts over several rounds, providing quantitative and qualitative feedback between rounds allowing experts an opportunity to rerate items based on other participants’ responses. ¹¹ This model is best suited for when consensus can be reached without requiring extensive discussion or clarifications. The Delphi method involves six stages: (1) identify a research problem, (2) conduct a literature search, (3) develop a questionnaire including the proposed list of statements, (4) conduct anonymous iterative questionnaire rounds (via email or mail), (5) provide individual and/or group feedback between rounds, and (6) summarize the findings. ² The unique feature of the Delphi technique is that the experts in their field selected for participation are to remain anonymous to one another, never meeting in‐person or interacting directly with the other experts in the panel. This is important to eliminate the bias and influence that can occur in face‐to‐face meetings, allowing experts’ opinions to be expressed freely without fear losing credibility or devolving into “group think.” ³ , ¹² , ¹³ Another benefit of the Delphi method is that the questionnaires and responses between rounds can be shared electronically and asynchronously, such that participants can respond at their own pace, avoiding the fatigue of a long meeting. ³ , ¹³ This can also allow much broader distribution to a large number of participants who may be geographically dispersed with minimal support structure needs and costs. ²

In contrast, a nominal group technique is preferable when a topic is poorly defined, and items in the questionnaire require considerable interpretation. ¹ , ¹³ The nominal group technique typically involves a select number of experts and stakeholders using a face‐to‐face meeting to provide an opportunity for discussion. There are six phases: (1) formulate and present the nominal question, (2) idea generation, (3) sharing of ideas, (4) structured discussion with a skilled moderator, (5) private ranking of ideas, and (6) sharing anonymous feedback with the group. ² During idea generation, the nominal group participants privately, and in silence, develop ideas or solutions to a given problem. They subsequently share the ideas in a round‐robin fashion. ¹ , ¹³ , ¹⁴ , ¹⁵ These ideas or solutions can be listed for sharing on a flip chart or in a virtual repository for the entire group. ¹⁴ , ¹⁶ , ¹⁷ In some instances the ideas may be shared with the moderator who subsequently shares with the group if they desire to preserve anonymity in the report‐out phase. ¹³ To reduce the risk of bias, the moderator should be skilled at facilitating group discussions and ensure that all participants are able to express their views while keeping particular personal and professional views from dominating the discussion. ¹ The facilitator can either be an expert on the subject or a nonexpert who has credibility with the participants. ¹⁸ The idea generation phase is considered a strength of the nominal group technique. Ideas may be generated de novo or, depending on the subject matter, derived from literature or other materials provided to participants. ¹ , ¹⁴ , ¹⁹ Each member must listen to each idea or solution and is not permitted to discuss them during this phase. ¹ , ¹³ , ¹⁴ In the third phase, a discussion is moderated, allowing clarification around the ideas or solutions presented. ¹ , ¹³ , ¹⁴ In the fourth phase, ideas or solutions are ranked by the participants anonymously. ¹ , ¹³ , ¹⁴ The core nuances of this approach are that the participants are afforded the ability to clarify ideas and solutions in real time and participants are kept on task and focused for the duration of the meeting, allowing for rapid attainment of consensus. ¹³ , ¹⁴

When choosing one consensus method over another, having the purpose in mind is crucial. Returning to our examples in Box 1 above, we can see how one technique may be more favorable than another based upon the specific scenario. For example, if you are attempting to determine the consensus present for item inclusion on a national objective structured clinical examination (OSCE) for graduating residents (Case #1), there will be limited need for item generation or group discussion and the emphasis will be on measuring the degree of agreement. Therefore, the Delphi technique will likely be preferable. Alternatively, if you are seeking a way to engage a group of local experts to develop and come to a consensus on topics for inclusion in an ultrasound curriculum for your medical school (Case #2), idea generation and discussion will be core needs. As such, the nominal group process may be the more optimal way to achieve consensus.

BOX 1. Case Examples.

Case #1:

You are the committee chair for developing a national objective structured clinical examination for graduating residents. The committee has decided on the content of the stations, but checklists need to be developed for each station that considers national standards. How will you develop these checklists?

Case #2:

You have been tasked with developing an ultrasound curriculum for undergraduate medical students in your medical school. After a literature search you have a few items but are not sure if it is comprehensive enough. How will you ensure you have considered all the appropriate topics and selected the most important ones?

2. Develop the initial questionnaire

Begin with a literature search to identify existing literature on the topic and prior consensus‐based documents in related fields or topics. This should include a structured search of major published databases as well as consideration of a search of the gray literature. Additional information may also be gathered from relevant end‐users who would not be involved in the process but have relevant experience or interest. ²⁰ Next, researchers must decide whether they plan to begin with an open‐ended list using the consensus process for the idea generation, to provide a predetermined list, or a combination of both. ²¹ As above, this should align with the intended purpose of the study. Once these initial questions have been developed, the list should be pilot tested for clarity and interpretation. When designing questions, researchers may also want to consider the length and complexity of the questions, as this may influence response rates. ³

In addition to the individual items, researchers must also consider the prompt used to assess consensus for each item. Consensus can be achieved by using either selecting statements (e.g., “must/definitely include,” “do not include”) or by rating their agreement with a given statement, such as using a Likert scale. We also recommend having an option for “inclusion with modifications,” so that participants can comment on suggested edits to improve the item or clarify language.

3. Identify the participants

When identifying participants for a consensus‐based technique, it is generally recommended to use topic experts. While many envision this to represent researchers on a given topic, we suggest considering a broader range of stakeholders or end‐users (i.e., those who will ultimately use the findings from the consensus project). ² , ²⁰ , ²² This can include researchers, educators, and even learners or patients who have an understanding of that topic. Panels should seek to ensure diversity among participants as this has been demonstrated to increase the breadth of ideas and considerations and may improve overall performance. ²³ , ²⁴ Although broad ranges of participants in a group may enhance creativity, it may be offset by increased task conflict in small groups. ²⁵ Additionally, it is important to ensure participants are reliable and can commit to the full study. Consensus methods require sustained involvement over time for serial rounds of surveys (and potentially real‐time meetings), so participants should be queried regarding their capacity to do so prior to engaging in the process.

Another important consideration is the number of participants to recruit. Nominal group technique typically uses 5 to 12 members, so as to ensure sufficient viewpoints while also allowing enough time for all voices to be heard. ²¹ In contrast, Delphi studies can range from a handful of participants to over a thousand. ⁸ Most Delphi studies use between 11 and 25 participants. ⁸ Rather than emphasizing a single specific number, we propose that researchers seek to balance the breadth needed for a variety of viewpoints with the risk of attrition and declining voice in larger groups.

4. Determine the number of rounds and consensus threshold

The iterative nature of consensus methods is one of the hallmarks of these methods. When reaching consensus, this design is meant to foster clarification and discussion, with each successive round providing feedback to the participants to aid in understanding or decision making. This often requires several rounds, and we suggest that two or three rounds is likely ideal for most studies. ²⁶ In a recent review of consensus methods, nearly 80% of these studies reported two or more rounds. ²¹ The number of rounds should be determined a priori and include a plan for how to approach items not reaching consensus within the specified number of rounds (e.g., automatic exclusion, supplemental rounds, topic to be researched). This will help inform decisions for subsequent rounds and guide participants as to the anticipated time commitment.

Even more important than the number of rounds is the consensus threshold. The consensus threshold is the value for which an item should be selected for inclusion (and, in some cases, selected for exclusion). There is substantive variability in practice, ranging from 51% to 100% agreement. ²¹ The median threshold for defining consensus in one review paper was found to be 75%. ⁸ Consistent with this, we recommend 70–80% as a reasonable threshold based on our experience. Some may also consider using a threshold for exclusion, where items are removed from further rounds when a set percentage of participants vote not to include the item. This offers the advantage of removing items and shortening the survey, which can be helpful to improve response rates. ²⁷ Ultimately, the threshold will vary depending on the purpose of the study, and many modifications are possible. ¹⁵

5. Describe and justify any modifications

The Delphi technique was initially developed in the 1950s to structure group communication to increase accuracy of forecasts by the RAND Air Force Corporation. ¹⁸ Since then, there has been an ever‐expanding list of modifications to the technique. Therefore, using the term “modified Delphi” is meaningless, and researchers must instead describe each step used and justify choices made. The justification is particularly important as there is often little to no empiric evidence to support the changes. ²⁸ For example, forcing consensus by restriction on discussion in a Delphi study may or may not be appropriate. ²⁹ Developing items for a curriculum may in fact warrant forcing consensus, but strong disagreement on checklist items for a national OSCE station may highlight differences in clinical practice which warrant future discussion. Hence, finding dissensus may be as important as developing consensus. Selection of expert participants also requires careful consideration. If using the Delphi to develop national entrustable professional activities for a specialty, researchers may want to consider geographical representation; equity, diversity, and inclusion; specialists at various stages of their career; patient representatives; and allied health professionals. While providing participants with quantitative feedback between rounds to see their score relative to others is a key feature of consensus methods, qualitative feedback (e.g., summary statements and item feedback) is not universal. Participants may see feedback from peers only (e.g., physicians), from multiple stakeholders separately (e.g., physicians and patients), or from multiple stakeholders combined which may or may not impact outcomes. ²⁸ Moreover, while the nominal group has traditionally been conducted in person, increased virtual conferencing options and pandemic restrictions have prompted some to convert to virtual models. ³⁰

In all of these considerations there is no “correct” answer, but researchers must clearly defend their choices in light of the study purpose, so the reader can consider the appropriateness and implications on the results. While Delphi reporting guidelines have been published, ²¹ , ³¹ an updated set of guidelines are currently being developed. ³²

CONCLUSIONS

In this article, we highlighted strategies within five key steps for ensuring best practices in consensus methods. These strategies include suggestions for deciding which consensus method is best, developing the initial questionnaire, identifying the participants, determining the number of rounds and consensus threshold, and describing and justifying any modifications. By following these strategies, education researchers can help ensure that their approach is rigorously conducted and reported.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Gottlieb M, Caretta‐Weyer H, Chan TM, Humphrey‐Murto S. Educator's blueprint: A primer on consensus methods in medical education research. AEM Educ Train. 2023;7:e10891. doi: 10.1002/aet2.10891