Online Modified-Delphi: a Potential Method for Continuous Patient Engagement Across Stages of Clinical Practice Guideline Development

Patient engagement has become a key component and indicator of high-quality development of clinical practice guidelines (CPGs).^1–5 Engaging patients in CPG development respects their expertise in the lived experience of clinical conditions and aims to create more patient-centered guideline recommendations.^{6, 7} As this area of guideline development has matured, guideline development groups and other healthcare policy stakeholders are interested in shifting from engagement of a limited number of patients at isolated phases of CPG development to more in-depth, continuous engagement of a larger group of patients across all stages of the development process.⁸ Frameworks for continuous patient engagement in CPG development have highlighted various approaches to identify the preferences and perspectives of patients, caregivers, and other patient representatives by using surveys, focus groups, and representation on guideline development groups.⁹

Online Delphi processes offer a promising approach to rigorous, systematic, scalable, and convenient engagement of patients in all stages of CPG development.¹⁰ The online Delphi process is an iterative, multistage method to solicit information and synthesize opinion into group consensus.¹¹ “Traditional” online Delphi processes involve a series of sequential questionnaires answered anonymously by a panel of stakeholders with relevant expertise on the topic of interest.¹² Online “modified-Delphi” processes combine survey rounds with a round of moderated, anonymous, online discussion boards.¹³ For example, ExpertLens^TM is an online system for conducting Delphi processes that automatically summarizes responses from each structured questionnaire round, provides these summaries to participants after each round via controlled feedback, and allows them to discuss results using anonymous online discussion boards in which participants post and respond to written comments.¹⁴ Combined, these features allow participants to consider their previous opinions, challenge others’ ideas, and stimulate engagement with new concepts without pressures to conform to other participants.

Online modified-Delphi processes are gaining traction as a patient engagement method due to several comparative advantages.^15–17 First, they include anonymous discussion boards and controlled feedback that reduce social-cognitive biases (e.g., yielding to real or perceived social pressure) more likely to lead to conformity in group discussion.¹¹ By using successive online surveys, controlled feedback, and anonymous discussions, participants may be more likely to consider group opinions in a non-adversarial manner compared to focus group formats.¹² Second, feedback and anonymous discussion that are moderated by the study team may help avoid group dominance by certain individuals and decrease pressures toward group conformity. Third, online modified-Delphi processes maintain a plurality of views through facilitated group interaction. Participants are still informed by the collective opinions of the group and can identify items that may have been missed or thought unimportant, allowing the opportunity for participants to change their opinions in light of feedback.¹¹ This structured process facilitates an efficient and effective way to solicit input from geographically dispersed stakeholders on important and potentially complex topics.¹³

In light of these advantages, online modified-Delphi processes can be used as a method for patient engagement in CPG development.¹⁰ For example, we developed an online modified-Delphiapproach—theRAND/ Parent Project Muscular Dystrophy Patient-Centeredness Method (RPM)—to engage patients and their caregivers in rating the patient-centeredness of CPG recommendations (see Text Box 1).^{7, 18} Based on findings from a proof-of-concept study, we developed practical step-by-step guidance on how online modified-Delphi process could be used to engage patients and caregivers in determining the patient-centeredness of CPG recommendations.²³ Upon reflection of our project, we believe that online modified-Delphi processes can be used throughout all stages of CPG development to continuously engage a panel of patients and their representatives. In this Perspective, we propose ways that guideline development groups can use online modified-Delphi processes to engage a large panel of patients in a manner compatible with best-practices at each stage of CPG development^{1–5, 24} and consistent with frameworks for continuous patient engagement across the entire CPG development process.⁹ Unless otherwise specified, “patients” refers to patients, caregivers, and their representatives. While we focus on online modified-Delphi processes, proposals below are also relevant to “traditional” online Delphi processes unless specific to the use of moderated discussion boards.

Text Box 1 The RAND/PPMDPatient-Centeredness Method^{7, 18–21}

ONLINE DELPHI PROCESSES AS A CONTINUOUS PATIENT ENGAGEMENT METHOD ACROSS CPG STAGES

At the beginning of the CPG development process, patients can help nominate and prioritize topics that they find important. Online modified-Delphi processes can be used to have patients systematically propose and develop consensus on the most important CPG topics. This process could either help identify an entire disease state that requires CPGs or decide which topics within that disease state to address. Specifically, patients could propose potential topics in an initial and open-ended “brainstorming” round, and then prioritize them during subsequent rounds using close-ended rating or ranking questions. Alternatively, an online modified-Delphi process could help engage patients in prioritizing an existing list of potential topics.

Following topic selection, patients provide a unique perspective on selecting guideline development group members to help ensure the group has a representative and trustworthy composition. Online modified-Delphi processes could be used to propose and prioritize desired characteristics of guideline development group members, including the type of expertise or experience group members should have. Given the importance of balancing conflicts of interest in guideline development groups,²⁵ patients also could indicate which conflicts of interest and mitigation strategies are most salient for developing a given CPG.

Once the guideline development group is established, patients can assist with developing the questions, analytic framework, and research plan for the evidence review to ensure that its scope will be relevant and useful to real-world issues that the CPG aims to address. Online Delphi processes have been used previously to finalize the questions and outcomes addressed in CPGs.²⁶ Specifically, patients can help refine clinical questions to be addressed through an online modified-Delphi process that develops consensus on the importance of sub-populations, outcomes, aspects of treatments, and other elements proposed for inclusion in the research questions, analytic framework, and research plan.²⁷

When forming conclusions about the literature and developing recommendations, patients can appraise the degree to which review summaries and conclusions seem valid, meaningful, and intelligible.²⁸ Patient engagement particularly can assist in developing recommendations that foster shared decision-making, respect variability in patient preferences, and identify gaps from a patient perspective.^{7, 19} Online modified-Delphi processes can be used to identify consensus on which review summaries and conclusions patients find most believable, credible, or useful. Guideline development groups can then engage patients in recommendation development to ensure that recommendations facilitate patient-centered care. For example, we developed the RPM specifically to engage large numbers of patients and caregivers in rating the patient-centeredness of CPG recommendations already deemed clinically appropriate and necessary.²⁰ Online modified-Delphi processes such as the RPM also can be used in the initial development of recommendations according to established methods for CPG development. For example, the RAND-UCLA Appropriateness Method is used by several groups to identify recommendations that clinicians find appropriate and necessary²²; rather than after this step, the RPM could be used concurrently to identify patient consensus on key aspects of evidence-to-decision frameworks,⁷ such as outcome importance, recommendation acceptability, and equity concerns related to disadvantaged patient sub-populations.²⁹

Once CPGs have been finalized, patients can then assist in disseminating and implementing CPG recommendations, such as facilitating the use of acceptable terminology in CPG recommendations and lay language summaries. For example, patients and patient advocacy groups can endorse guidelines to improve their legitimacy and trustworthiness to patient communities, as well as patient adherence to guideline recommendations. Online modified-Delphi processes can be used to engage patients in identifying items to include in decision aids, as well as barriers and solutions to guideline uptake.¹⁹

Lastly, patients can assist in updating and evaluating CPGs, such as determining if and when guidelines require an update, assessing whether patients made meaningful contributions to a CPG, and informing improvements to future patient engagement strategies in CPG development. To facilitate patient engagement at this stage, online modified-Delphi processes can be used to continually assess the current patient-centeredness of CPG recommendations. For example, patients can define and prioritize significant changes in views from the patient community since a CPG has been developed, such as outcomes and interventions that should be included in a CPG update.

RESEARCH AGENDA FOR DELPHI-BASED CONTINUOUS PATIENT ENGAGEMENT IN CPG DEVELOPMENT

We propose several avenues of future research to investigate empirically the potential of using online modified-Delphi processes to continuously engage patients in CPG development. First, future research could directly compare online modified-Delphi processes with other engagement methods to examine comparative advantages to (or synergistic benefits when used in conjunction with) other methods for continuous patient engagement in CPG development.^{9, 10} For example, future research could examine the value of online modified-Delphi processes that allow patients to interact with each other, reflect on their views in light of feedback from their peers, and explore consensus over iterative rounds by comparing their outcomes to those of online surveys. In addition, future research could test whether the benefits of online modified-Delphi processes over face-to-faceinteractions—such as groupthink, being overruled by traditional experts or professionals, and tokenistic representation—replicate when used with patients in the context of CPG development.^{11, 12, 14, 30, 31} For example, guideline development groups could examine how much time and resources are needed to ensure patients understand the task and language in online modified-Delphi processes compared to in-person methods that allow for live explanation and training.^{32, 33} Moreover, empirical studies could examine how online modified-Delphi processes can be used in combination with other engagement strategies to increase the representation of patient views in CPG recommendations.

Recruitment and retention procedures are another important avenue of future research. A major potential advantage of online modified-Delphi processes is their ability to engage a large number of patients with diverse perspectives, despite geographic and transportation barriers.³⁴ As a result, they could enhance representativeness of the target patient population and sub-populations, particularly groups historically underrepresented or marginalized in the development of healthcare policy and practice. To achieve this aim, effective recruitment and retention are essential. While the ability to accommodate a large number of participants is a benefit, it also means guideline development groups may have to spend significant time and resources recruiting and retaining these patients.³⁵ Patients may not have sufficient time, interest, or resources themselves to complete multiple Delphi rounds. Future research could examine strategies we found useful in our RPM pilot project, such as reviewing previous research on patient preferences, using an established and curated patient registry, purposively sampling a diverse group of patients (e.g., demographically, geographically, and in stages of disease progression), and tactics to keep participants engaged across rounds (e.g., personalized reminders, incentives for each round, flexible deadlines).²³

Lastly, future research should focus on the conditions under which patient engagement with online modified-Delphi processes is most beneficial. For example, while they could be used to approve proposed members of the guideline development group, this approach could lead to interpersonal tension if patients are asked to rate directly their approval of specific individuals. Moreover, patients may require more training for more-technical stages and aspects of CPG development (e.g., evidence review methodology). In contrast, examples exist of using online Delphi processes as part of finalizing or validating products already drafted by the guideline development group, such as CPG recommendations,³⁶ quality indicators based on guideline recommendations,³⁷ and patient summaries.³⁸ In addition to specific stages of CPG development, continuous patient engagement with online modified-Delphi processes also may vary in utility based on the remit of a given CPG. For example, our RPM pilot focused on care considerations for a genetic disorder with a pediatric onset and with which patients and caregivers have significant lived experience; a CPG focusing entirely on screening and prevention decisions for asymptomatic patient populations may face greater difficulty in sustained, continuous online patient engagement. Future research examining these issues ultimately could shed light on the potential of continuously engaging patients via online modified-Delphi processes to enhance the trustworthiness, credibility, utility, and patient-centeredness of CPG recommendations.

Funding

This work was supported through a Patient-Centered Outcomes Research Institute (PCORI) Program Award (ME-1507-31052). All statements in this article, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, its Board of Governors or Methodology Committee, or the official position of the Centers for Disease Control and Prevention. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.

Declarations

Conflict of Interest

DK and SG are members of the RAND ExpertLens team. SG’s spouse is a salaried employee of Eli Lilly and Company and owns stock. SG has accompanied his spouse on company-sponsored travel. All other co-authors report no conflicts of interest.