Modified Delphi survey for the evidence summarisation of patient decision aids: Study protocol

Abstract

Introduction

Information included in a patient decision aid (PDA) can significantly influence patients’ decisions and is, therefore, expected to be evidence-based and rigorously selected and summarised. PDA developers have not yet agreed on a standardised process for the selection and summarisation of the supporting evidence. We intend to generate consensus on a process (and related steps and criteria) for selecting and summarising evidence for PDAs using a modified Delphi survey.

Methods and analysis

We will develop an evidence summarisation process specific to PDA development by using a consensus-based Delphi approach, surveying international experts and stakeholders with two to three rounds. To increase generalisability and acceptability, we will distribute the survey to the following stakeholder groups: PDA developers, researchers with expertise in shared decision making, PDA development and evidence summarisation, members of the International Patient Decision Aids Standards (IPDAS) collaboration, policy makers with expertise in PDA certification and patient stakeholder groups. For each criterion, if at least 80% of survey participants rank the criterion as most important/least important, we will consider that consensus has been achieved.

Ethics and dissemination

It is critical for PDAs to have accurate and trustworthy evidence-based information about the risks and benefits of health treatments and tests, as these decision aids help patients make important choices. We want to generate consensus on an approach for selecting and summarising the evidence included in PDAs, which can be widely implemented by PDA developers. Dartmouth College’s Committee for the Protection of Human Subjects approved this protocol. We will publish our results in a peer reviewed journal.

Strengths and limitations of this study.

• Gap: There is no standardised method for selecting and summarising the evidence in patient decision aids (PDAs).

• Solution: We are developing a process to ensure PDAs have the most up-to-date, trustworthy evidence available.

• Clinical implications: This will help patients and clinicians know they can trust the information in PDAs so they can make the best decisions together.

• Strengths: Systematic involvement of patient stakeholders.

• Limitations: Limitations of online surveys include selection bias.

Introduction

Patient decision aids (PDAs) are tools that help patients and their clinicians make preference-sensitive decisions together. They are typically defined as ‘evidence‐based tools designed to help patients make specific and deliberated choices among healthcare options. PDAs supplement (rather than replace) clinicians' counselling about options’.^{1 2} They promote patient engagement in medical decision-making, collaboration between patients and their care team, increase knowledge and align patients’ choices with their preferences.¹ Therefore, the information included in PDAs can significantly impact patients’ decisions. For this reason, patients and clinicians expect the information in PDAs to be evidence-based and rigorously selected and summarised.

The approach that PDA developers use to select and summarise the evidence in PDAs, however, appears inconsistent. A recent international cross-sectional survey of 15 PDA developers confirms that they do not have an agreed-upon, standardised process to select and summarise evidence. They also do not always document the evidence selection and summarisation process.³ Most organisations reported using existing systematic reviews and clinical practice guidelines to select and summarise information for PDAs. Less than half reported using a standard, documented approach to guide the evidence selection and summarisation. When the approach was documented, the documents offered varying levels of detail. Common evidence summarisation steps identified were: tool-relevant question formation, search strategies, evidence appraisals and updating policies. There was no standardised process across organisations to summarise evidence for PDAs. Although agreed-upon approaches and tested methods for evidence summarisation, such as clinical practice guidelines, exist in other areas, there is no agreed process (including steps and criteria within each step) for the selection and summarisation of evidence for PDAs.

The International Patient Decision Aids Standards (IPDAS) collaboration developed criteria for assessing the quality of PDAs.⁴ These criteria are also used by PDA producers to guide the development of the interventions. However, only six items of the IPDAS checklist cover the selection and summarisation of evidence, and do not provide any guidance about recommended methods for evidence selection and summarisation of PDAs.⁴ A 2013 review of the literature conducted by the IPDAS working group on the synthesis of scientific evidence highlighted the importance of rigorously selecting and summarising evidence used to populate a PDA. They did not provide clear practical guidance on how to conduct evidence summarisation for the development of PDAs except recommending that developers apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology.⁵ Further, the IPDAS instrument and the IPDAS minimum standards do not offer additional information or guidance on the steps required to select and summarise evidence-based information for PDAs.^{6 7} Other efforts to evaluate or certify the quality of PDAs have emerged,⁸ but none of those standards or certification bodies describe recommended methods and criteria that PDA producers should follow when selecting and summarising evidence for patient-facing interventions.

Evidence summarisation in other medical contexts, such as the selection and summariation of evidence for clinical practice guidelines and systematic reviews, is increasingly standardised. This process promotes transparency, rigour and minimises the risk of bias in the end product.^{2 9–17} The same level of scrutiny is justified when developing PDAs, as they may directly influence patient care and decision-making. Tasks such as the selection and identification of patient-relevant outcomes, analysis of patient concerns and priorities, description of the quality of evidence and communication of uncertainty in ways that patients understand, warrants the development of an agreed process and related steps and criteria that are specific to PDAs. For these reasons, it would not be appropriate to apply evidence summarisation processes developed for clinical guidelines without integrating the evidence summarisation steps and components that are specific to the development of interventions that target patients. The target group, scope and content differ significantly enough from clinical practice guidelines development, thus requiring a tailored evidence summarisation process. Additionally, the IPDAS impose some prerequisites on the evidence summarisation process on which the decision aid will be based. For example, IPDAS requires that the decision aid summarise the evidence regarding all health options available to a patient facing a specific health problem, and that decision aids present positive and negative features of each option with equal amount of details, among other specificities.¹⁸ Efforts to develop an agreed evidence summarisation process for PDAs should incorporate the substantial body of related evidence summarisation guidance previously developed by other groups, and notably for clinical practice guidelines previously mentioned.¹¹

Objective

The purpose of the study is to generate consensus on a process (and related steps and criteria) for selecting and summarising evidence for PDAs using a modified Delphi survey. This will in turn improve transparency, rigour and minimise the risk of bias of the evidence summarisation processes leading to the development of PDAs.

Methods

Study design and procedures

We will develop an evidence summarisation process specific to PDA development by using a consensus-based Delphi approach previously used in the development of a quality criteria framework for PDAs.^{2 19} Consensus methods can harness the views of international experts on a wide range of information and questions in order to make decisions that are based on expert consensus.²⁰ We will conduct a multi-round modified Delphi survey (two to three rounds). Compared with the nominal group technique, it is the most practical and scalable method to obtain feedback from a large number of stakeholders in different geographical locations. During the multiple rounds of online questionnaires, relevant stakeholders will be consulted to provide feedback about the evolving set of evidence summarisation steps and criteria. The anonymous responses from participants will be fed back to them in subsequent rounds. Depending on the level of consensus after two rounds (see Data analysis section), we will determine whether to conduct a third survey round.

Study management

To oversee the tasks of (1) generating an initial set of criteria for the Delphi process and (2) managing the Delphi survey distribution and analysis, we convened a steering group. This group will oversee the project and will make strategic decisions about the study design, data collection and analysis processes, as well as agree on a final process and related set of steps and criteria. An invitation to join this group was posted on social media (Shared@Shared Decision Making Network Facebook group: 745 members) on 30 June 2017. The post invited all Facebook group members to join an in-person meeting about evidence summarisation during the International Shared Decision Making conference held in Lyon, France, between 2 July and 5 July 2017. For those who were not able to join the meeting but expressed an interest in evidence summarisation of PDAs, a high-level summary was posted on Facebook. The steering group was convened in September 2017. The study steering group includes nine international experts in PDA development, evaluation and implementation, evidence summarisation and clinical practice guidelines, and one patient representative. Six steering group members are based in the USA, one in Canada, one in Australia and one in Spain. Google Drive and video-conferencing facilities will be used to facilitate the exchange and review of information and documents, virtual meetings, as well as real-time collaboration and version-control.

Participants

To maximise the generalisability and applicability of the criteria, we plan to invite for participation in the survey members of the following groups: (1) all known developers of PDAs who created or updated a tool within the last five calendar years (using existing inventory), (2) all members of the of the IPDAS group, (3) the shared decision making listserv, (4) the Society for Participatory Medicine listserv, (5) an overdiagnosis Google group, (6) the evidence-based healthcare listserv, (7) the Society for Medical Decision Making, (8) the Society of Behavioural Medicine (Health Decision Making Interest Group), (9) HTAi-ISG patient involvement listserv, (10) GRADE working group, (11) the Guidelines International Network, (12) convenience sample of policy makers with interest and expertise in PDA certification, (13) the BMJ patient group and (14) the ProPublica Patient Safety Community. We have no other eligibility criteria (except for membership of one of the above listed groups).

For all participants, the survey invitation (supplementary file 1) will provide a brief outline of the study, a link to the online survey (supplementary file 2), and a brief participant information sheet as the first page of the survey. Consent will be inferred by participants’ completion of the survey. The ethics application form and protocol were submitted to Dartmouth College’s committee for the protection of human subjects on 27 April 2018. Approval was granted on 23 May 2018 (STUDY00031042).

In order not to contaminate the Delphi survey results and express their views twice (in developing the original items and taking the surveys), the steering group members have unanimously decided not to complete the Delphi surveys.

Patient and public involvement

Design

Our patient partner, SC, was involved in the development of the Delphi survey and provided meaningful feedback on iterative drafts of the online questionnaire. SC is a core member of our study steering group and an author on this manuscript.

Participants

We also plan to make a concerted effort to recruit patient participants. We will reach out to online patient groups, including the BMJ patient group and the ProPublica Patient Safety Community (more than 6000 members). We will also engage a patient and family advisor group at Dartmouth-Hitchcock Medical Center.

Analysis

Our patient partner will be a critical part of our analysis team, and will be involved in all steering group meetings.

Survey development

The main output of the original Lyon evidence summarisation meeting was the creation of a spreadsheet that detailed all evidence-summarisation steps inherent to PDA development. The first draft of this spreadsheet, iteratively developed by the steering group members, included 18 criteria. Combining those 18 criteria with the eight existing standards for the summarisation of clinical practice guidelines as outlined by the National Academy of Medicine (formerly IOM) & US Preventive Services Task Force Standards led to the creation of the first draft of the proposed process and steps. This draft was shared in a Google doc with all members of the steering group and iteratively refined and finalised. Three separate iterations of the process (phases, steps and criteria) were created, reviewed and discussed by the steering group members until no additional revisions were suggested. A final internal version of the criteria (n=48), categorised into four phases and 13 steps was finalised in April 2018 (see supplementary file 3).

Data collection

Round 1 survey

The round 1 survey will include a brief information page and a summary of the process that led to the development of the phases, steps and criteria. Participants will be asked to provide their input on the phases, steps and criteria (including inclusion, wording, grouping, order and any other comments). Specifically, they will be asked to indicate, using a four-point Likert Scale (omit, possible, desirable, essential), whether each criterion included in the proposed process should be omitted or kept (and whether it is considered possible, desirable or essential). The criteria will be grouped into relevant phases and steps. For each phase and for each step, participants will be given the opportunity to provide rewording suggestions, suggest additional phases, steps or criteria, comment on the order of those elements or provide additional comments or questions. Email addresses will be collected so that participants can participate in further rounds. At the end of each round, we will confirm participants' interest to participate in the next round. Participants will also be asked to complete basic demographic questions. Each round of the survey will be open for 3 weeks, and two reminders will be sent.

Round 2 survey (and round 3, as necessary)

Round 1 participants will be invited to complete a second survey, in which feedback will be provided about the results of the first round (percentage of participants who thought a criterion should be included or excluded) and about the changes made based on the qualitative feedback. Participants will be invited to indicate whether to omit or include (omit, possible, desirable, essential) the items, including the new items proposed by participants in the first round, and to provide additional rewording suggestions, comments or questions. As mentioned above, the survey will be open for 3 weeks, and two email reminders will be sent. Depending on the level of consensus (see data analysis section), a third round may be conducted. This will be determined by the steering group after round 2 data analysis is completed. We will use open debate and discussion followed by a democratic consensus.

Data analysis

Following round 1, the ratings will be summarised using percentages and the views of all participants will be given equal weight. If at least 80% of participants rate an item in the lower two categories (omit, possible) or in the higher two categories (desirable, essential), we will consider consensus as having been achieved and the item will be removed or retained, respectively. Items where ratings do not meet the consensus threshold and conflict with open text comments will be grouped together and explained to round 2 participants. They will be asked to re-rate those items taking the qualitative feedback into account. Following the first survey round, a consensus meeting involving the steering group will be held. The steering group will review and discuss the ratings and qualitative feedback received, including rewording suggestions per criterion, suggestions to add new phases, steps or criteria and more general comments or questions. The wording or order of the phases, steps or criteria will be revised if two or more respondents suggest it or if the steering group members agree that the phase, step or criterion would benefit from rewording, reordering or merging.

Following the second survey round, a second consensus meeting will be held. Decisions on whether to conduct a third round and retain items in the Scale will be made based on the ratings in the survey rounds and feedback/comments from participants. The ratings will be summarised using percentages and the views of all participants will be given equal weight. If at least 80% of participants rate the importance of the item in the lower two categories, or in the higher two categories, we will consider consensus as having been achieved and the item will be removed or retained, respectively. If no consensus is achieved or the consensus ratings are contradicted by recurring open text comments, the steering group will decide whether or not to retain a criterion, basing this decision on qualitative feedback from the participants where possible, and the steering group’s views. We have successfully used this approach before.²¹

Only complete surveys will be included in the analysis. We will report the amount of missing data in the manuscript reporting the results of the Delphi survey.

Data management and safety

Data to be collected include information about the participant’s role as it relates to PDAs, general demographics and their opinion of what to add/change/include in an evidence summarisation process. We are careful to protect the identity of all study participants. We will store the data securely in accordance with standard human subject research protocols. All data will be retained for 3 years, per the Dartmouth College data retention policy (or for the period specified by journals in which arising manuscripts are published, if longer) and then destroyed securely.

Discussion

PDAs must have accurate and trustworthy evidence-based information about the risks and benefits of health treatments and tests, as these tools help patients make important healthcare choices. We want to generate consensus on an approach for selecting and summarising the evidence included in PDAs, which we hope can be widely adopted by decision aid developers.

Strengths and limitations

A strength of this study is the systematic involvement of patients and relevant stakeholders in planning the modified Delphi survey. We plan to include a diverse sample of participant stakeholders including patients, researchers, PDA developers and health policy makers. Limitations of online surveys always include the possibility of selection biases, meaning participants who opt to take the survey may be systematically different from the target population. In our case, the participants may be more engaged and more interested in the outcome of the Delphi survey. There is also a possibility that their views will be stronger than those who opted not to participate.

Conclusion

Patients should be able to trust the information they receive from PDAs. Together with their clinicians, family and caregivers, they rely on these tools to make decisions that are aligned with their informed preferences. We believe standardising a process for selecting and summarising the evidence included in PDAs is therefore a worthwhile effort. Bringing all relevant stakeholders to the table - patients, researchers, PDA developers and healthcare policy makers - will ensure that the ultimate outcome is rigorous and rooted in consensus to promote widespread adoption.

Ethics and dissemination

We plan to publish our results in a peer-reviewed journal.