Systematic Development of Patient Decision Aids

Abstract

Background.

The 2013 update of the evidence informing the quality dimensions behind the International Patient Decision Aid Standards (IPDAS) offered a model process for developers of patient decision aids.

Objective.

To summarize and update the evidence used to inform the systematic development of patient decision aids from the IPDAS Collaboration.

Methods.

To provide further details about design and development methods, we summarized findings from a subgroup (n=283 patient decision aid projects) in a recent systematic review of user involvement by Vaisson and colleagues. Using a new measure of user-centeredness (UCD-11), we then rated the degree of user-centeredness reported in 66 articles describing patient decision aid development and citing the 2013 IPDAS update on systematic development. We contacted the 66 articles’ authors to request their self-reports of UCD-11 items.

Results.

The 283 development processes varied substantially from minimal iteration cycles to more complex processes, with multiple iterations, needs assessments, and extensive involvement of end users. We summarized minimal, medium, and maximal processes from the data. Authors of 54/66 articles (82%) provided self-reported UCD-11 ratings. Self-reported scores were significantly higher than reviewer ratings (reviewers: mean = 6.45 [SD=3.10]; authors: mean = 9.62 [SD=1.16], P<.001).

Conclusions.

Decision aid developers have embraced principles of user-centered design in development of patient decision aids while also under-reporting aspects of user involvement in publications about their tools. Templates may reduce the need for extensive development and new approaches for rapid development of aids have been proposed when a more detailed approach is not feasible. We provide empirically-derived benchmark processes and a reporting checklist to support developers in more fully describing their development processes.

Introduction

Any new patient decision aid needs to be created in some way. Existing design and development processes for patient decision aids vary significantly. Previous versions of the International Patient Decision Aid Standards (IPDAS) began to map this diversity, with the 2013 update offering a model process based on the authors’ review of the literature.(1) Since that last update, there has been increasing recognition of the importance of engaging the users of decision aids in their development. In fact, involvement of the end users in developing decision aids, such as patients and caregivers, as well as health professionals, is seen as essential to successful implementation of these tools in clinical care.(2) Vaisson and colleagues published an extensive systematic review (covering 607 articles published up to June 2017) of the design and development processes of patient decision aids, organizing data within a framework of user-centered design.(3) More broadly, patient partnership, meaning patients and families being full members of research teams, has gained traction as a way to better ensure the relevance and impact of health research.(4)

Current Standards for Decision Aid Development and Reporting

The original version of the IPDAS Standards and Checklist emphasized a careful development process that was documented and systematically applied (http://ipdas.ohri.ca/IPDAS_checklist.pdf). Table 1 provides criteria relevant to use of a systematic development process from the IPDAS Checklist.

Table 1.

IPDAS Checklist Criteria Related to Use of a Systematic Development Process.

Development Process: Does the patient decision aid have a systematic development process?

Includes developers’ credentials/qualifications

Finds out what users [patients, practitioners] need to discuss options

Has peer review by patient/professional experts not involved in development and field testing

Is field tested with users [patients facing the decision; practitioners presenting options

The field tests with users [patients, practitioners] show the patient decision aid is:

acceptable

balanced for undecided patients

understood by those with limited reading skills

In an update of the quality dimensions used to inform the IPDAS standards, Coulter et al.,(1) examined the relevance of a systematic development process and its application to published randomized trials of decision aids, associated development reports, and processes used by organizations experienced in developing patient decision aids. They identified the following key common features of the decision aid development process: a) scoping and design; b) development of a prototype; c) iterative “alpha” testing with patients and health professionals; d) “beta” or field testing in real world settings; and e) production of the final version for dissemination or further research. A model development process was also offered. Finally, they emphasized the importance of referencing the scientific evidence used to guide the aid content.

From their review, Coulter et al.,(1) concluded the following: 1) only about half of patient decision aids had been field tested with patients; 2) fewer had been field tested with health professionals not involved in the development process; 3) few described the methods used for reviewing and synthesizing the clinical evidence, and 4) trial reports were not a good source of information about development processes. At that time, no revisions or new standards were offered for the IPDAS Checklist to further guide developers of patient decision aids.

In 2018, the IPDAS Collaboration published reporting standards and a checklist for patient decision aid evaluation studies (Standards for UNiversal reporting of Decision Aid Evaluations, SUNDAE Checklist).(5) The SUNDAE Checklist includes an item calling for the development process for the patient decision aid to be described in the Methods section of the publication. The checklist item #7 reads as follows: “Briefly describe the development process for the patient decision aid (and any comparator), or cite other documents that describe the development process. At a minimum including the following: a) participation of stakeholders in its development; 2) the process for gathering, selecting and appraising evidence to inform its content; and 3) any testing that was done.” Yet, there remains concern that decision aid development processes are highly variable and often have little or no involvement of stakeholders other than researchers on the development team. Greater specificity regarding involvement of multiple stakeholders is needed.

Theoretical Rationale for a Systematic Development Process

Since publication of the 2013 article by Coulter and colleagues,(1) there has been a dramatic increase in the production and evaluation of new patient decision aids across a wide range of decision-making contexts. Coulter and colleagues articulated the importance of a systematic development process as follows: “It is important that [patient decision aids] are carefully developed, user-tested, and open to scrutiny, with a well-documented and systematically applied development process. Some decision aids have been designed for one-off use in studies to advance knowledge, while others are intended for wider use in a range of real-life clinical settings.”(1) Users of decision aids deserve assurance that the processes employed to develop these tools follow acceptable standards. Without application of standards, poor quality decision aids have the potential to cause harm by leading to uninformed decisions and decisions that do not align with patients’ values, which undermine shared decision-making. Documentation is required to support the trustworthiness, acceptability, utility, validity and reliability of the development process, and to demonstrate involvement of patients, health professionals, and other experts before adoption of the tools. Health professionals’ agreement with use of a decision aid, serving as champions on the development teams, and engaging meaningfully throughout the development process are seen as enablers of successful implementation in clinical practice.(2, 6)

The IPDAS Collaboration recently undertook an update of the concepts, theories, and evidence used to inform decision aid standards. This paper builds on previous work by the collaboration on use of a systematic development process for decision aids. In this update, we aimed to 1) provide further details and guidance about design and development methods that build on the empirical evidence presented in the last version of the International Patient Decision Aid Standards, 2) describe how principles of user-centered design have been applied during the development of decision aids through use of a new descriptive measure of user-centeredness (UCD-11)(7), and 3) propose a new reporting checklist that patient decision aid developers can use to report their development processes.

Throughout this paper, for brevity, we adopt terminology from the papers from which we drew data. Specifically, when we refer to ‘users’ of patient decision aids, this means patients, caregivers, family members and surrogate decision makers as appropriate to the decision context. Health professionals may also use patient decision aids in collaboration with patients and others; however, they are not the people whose health or family life is the focus of the tool. In this paper, we refer to health professionals as ‘health professionals.’ When referring to both groups together, we refer to ‘users and health professionals.’

Methods

Overview and Conceptual Framework

To provide further details about design and development methods (objective 1), we summarized findings from Vaisson and colleagues extensive systematic review of literature published up to June 2017. To describe the application of user-centered design during the development of decision aids (objective 2), we conducted a focused review of patient decision aid literature citing the previous update of this chapter, had analysts score the design and development processes, asked authors to self-report their scores, and compared the analyst-assessed scores to authors’ self-reports. To provide new reporting guidance (objective 3), we consulted with experts in the field and built upon the measure to provide a set of suggested elements to report in the form of a checklist.

This update is guided by recognition of the essential role users play in the development and successful implementation of patient decision aids. Vaisson et al.,(3) synthesized key literature in providing a framework for user-centered design that can be applied to patient decision aids. As seen in Figure 1, the design framework includes three elements involved in iterative cycles of development: 1) understanding the needs and context of users (and, if applicable, health professionals), 2) developing and refining the protocol, and 3) observing users (and, if applicable, health professionals) interact with the prototype. Users’ and health professionals’ needs can be assessed in a variety of ways, including literature reviews and collection of primary data through interviews, surveys and/or observations. These activities correspond to assembling a steering committee, scoping, and design steps from the model development process offered by Coulter et al.(1) Development and refinement of the prototype includes typical steps related to selecting the format of the aid, review of content, and storyboarding. These activities correspond to the prototype step from Coulter et al’s model.(1) Finally, observation of users’ and health professionals’ interactions with the aid correspond to alpha testing (with patients who have faced the decision and health professionals who treat them) and beta testing, or a field test, where the aid is used in real-world settings with patients facing the decision and their health professionals. These methods may be complemented by other observational research methods such as questionnaires to evaluate the aid’s usability and feasibility along with users’ and health professionals’ satisfaction.

Figure 1.

/Model development process incorporating user-centered design framework from the IPDAS collaboration.

Empirical Evidence about the Design and Development of Patient Decision Aids

To summarize empirical evidence, we used data from Vaisson and colleagues’ systematic review of the design and development of 325 patient decision aids, of which 283 had published data about the design and development processes.(3) In their review, Vaisson and colleagues noted considerable variability in the processes used by developers for understanding the needs of users and health professionals, reporting on development and refinement decision aid prototypes, and observing users and health professionals interacting with prototypes. Such variability may reflect differences in the needs for user and health professional involvement across different contexts, different methods used, variable attention and resources devoted to user involvement, or simply differences in reporting. The authors identified three key opportunities for improving the user-centeredness of patient decision aid development: 1) involving users and health professionals early in the development process to understand needs, limitations, etc.; 2) examining users’ and health professionals’ interactions with versions of the decision aid; and 3) reporting changes between iterative cycles of refinement and testing. They concluded with recommendations for co-design of prototypes involving patients, health professionals, and other relevant stakeholders. For readers interested in further details, we refer them to Vaisson and colleagues’ paper.(3)

For this update, we synthesized and organized the data from Vaisson’s systematic review to present three possible design and development processes: a minimal process, a medium process, and a maximal process. We defined a minimal process as conducting steps reported in the large majority (>70%) of projects from the systematic review, meaning that these could be considered basic requirements. When counts were available; for example, the number of users involved, we defined a minimal process as having counts at the first quartile. We defined a medium process as the minimal process plus conducting steps reported in many (40–69%) projects, with counts at the median. We defined a maximal process as the medium process plus conducting steps reported in a minority (5–39%) of projects, with counts at the third quartile.

Application of User-Centeredness in the Design and Reporting of Patient Decision Aids

We used a new measure of user-centeredness to describe how developers of patient decision aids published since the last update of the IPDAS standards (8) have applied principles of user-centered design in developing and reporting about their aids. The UCD-11 is an 11-item, quantitative measure of the user-centeredness of the design and development of patient-centered tools, including patient decision aids. Developed by Witteman and colleagues from published descriptions of the design and development processes of 348 personal health tools (of which 283 were patient decision aids; these were the same 283 as in Vaisson and colleagues’ systematic review), the UCD-11 contains items that address involvement of users at every stage of the user-centeredness framework, including iterative cycles of design and retesting (see Figure 1). The scale has acceptable internal consistency reliability, good item discrimination between low and high overall scores on the measure, and yields a single overall score.(3) Three broad activities are addressed in the measure: 1) pre-prototype involvement of potential end users, 2) iterative responsiveness involving cycles of testing and refinement, and 3) involvement of other experts in development, such as health professionals. Scores range from 0 to 11, with higher scores indicating greater user-centeredness of the design and development process. Patient decision aids in the data set used to create the measure had a median score of 5 out of 11 (interquartile range: IQR 3–8).(7)

To describe the application of user-centeredness in decision aid development, we first conducted a citation search using Web of Science in October 2019 to identify articles in which authors had cited Coulter and colleagues’ article on patient decision aid development.(1) Our results rendered 112 citations; we screened abstracts and titles to exclude articles not discussing the development of a patient decision aid. We included a final sample of 54 articles in the comparison of analyst and author ratings (Appendix Figure 1). Appendix Table 1 contains a list of the articles included in the final sample.

We selected five articles as a training set and two authors (KGM, RJV) read and scored these articles; these responses were available as examples for other reviewers. Other reviewers consulted these for orientation to the task and guidance. In an effort to obtain a sample that represented shifts over time, one article was selected from each publication year (e.g., 2014–2019). From there, the training set was selected with an effort to diversify the health context of the decision aid. Specifically, we did not include the same topic multiple times even if articles were published in different years. To reduce burden on all reviewers, we selected two articles for assessing inter-rater reliability. All reviewers received instructions as well as examples from the training set, with all reviewers completing the UCD-11 for two of the selected articles. We calculated interrater reliability using intraclass correlation coefficients (ICC) based on a mean rating (k = 5), absolute-agreement, 2-way mixed-effects model. The results showed good interrater reliability (ICC = 0.82, 95% CI: 0.62, 0.94). After completing the training set, reviewers were randomly assigned a subset of articles to score independently. Two reviewers who speak English as a second language elected to independently score both of their sets of articles, discuss any points of confusion, and reached consensus on the scores for those articles. The reviewers’ UCD-11 ratings were completed by June 2020.

We collected data online using REDCap.(9, 10) We calculated the frequency and percentage of each UCD-11 item endorsed as “yes”, as well as means and standard deviations for the total scores (sum of each item endorsed “yes”.) We compared differences between reviewers’ and authors’ ratings using a paired-samples t-test. Finally, we examined UCD-11 scores by publication year (2014–2019) to explore any change in application of user-centeredness over time.

We contacted the corresponding authors for each of the included articles and asked them to complete the UCD-11. This allowed the comparison of the authors’ known development process with the independent ratings of reviewers based on the publication(s) related to the patient decision aid development. Initial requests to authors of articles included in the review were sent in December 2019. We contacted authors up to three times, with final requests sent in February 2020. We created a REDCap database for data collection. We analyzed the data using SPSS version 24.

Reporting Checklist

We developed a reporting checklist based on UCD-11, expanding the description beyond a binary yes/no response of whether or not a step was conducted to include details of what the authors did in their study and adding items identified as important by members of the original UCD-11 development team, Vaisson and colleagues’ study team, and 15 external experts with experience developing different kinds of patient decision aids.

Results

Empirical Evidence

As shown in Table 2, among 283 patient decision aid projects from the review by Vaisson and colleagues, there was a wide range of design and development processes.

Table 2.

Minimal, Medium, and Maximal Patient Decision Aid Design and Development Processes Derived from 283 Projects.

	Minimal Process* (>70% of 283 projects, first quartile of counts)	Medium Process* (40–69% of 283 projects, median of counts)	Maximal Process* (5–39% of 283 projects, third quartile of counts)
Iterative nature of overall design and development process	• Process is iterative, with at least 2 cycles.** • Changes between cycles are not explicitly noted or reported.	• Process is iterative, with at least 3 cycles. • Changes between cycles are not explicitly noted or reported.	• Process is iterative, with at least 4 cycles. • Changes between cycles are explicitly noted and reported.
Development steps for understanding users (patients, family members, caregivers, surrogates)*** and their contexts	• Conduct literature review	• Conduct literature review • Conduct informal needs assessment*** with 30 users	• Conduct literature review • Conduct informal needs assessment with 43 users • Conduct formal needs assessment**** with 44 users • Observe existing processes (e.g., ethnography) with 56 users
Development steps for developing & refining prototype patient decision aid	• Develop prototype	• Conduct content and format review prior to prototyping with 14 users • Develop prototype	• Develop and/or validate underlying model • Storyboard or wireframe design • Adapt or translate content and format for different cultural groups with 38 users • Conduct content and format review prior to prototyping with 25 users • Develop prototype
Development steps for observing users’ (patients, family members, caregivers, surrogates) interactions with the prototype	• Conduct pilot or usability testing with 15 users	• Review content and format of developed prototype with 20 users • Conduct pilot or usability testing with 28 users	• Review content and format of developed prototype with 30 users • Conduct pilot or usability testing with 45 users • Conduct additional rounds of pilot or usability testing with 40 users
People involved as users (patients, family members, caregivers, surrogates)	• People currently facing this decision	• People currently facing this decision	• People currently facing this decision • People who faced this decision in the past • People who may face the decision in the future • People who are members of populations marginalized by social norms and policies are specifically involved
What is evaluated?	• Efficacy	• Efficacy • One or more of feasibility, acceptability, satisfaction, and usability	• Efficacy • One or more of feasibility, acceptability, satisfaction, and usability • Implementation
How is evaluation done?	• Users are asked their thoughts & opinions of the patient decision aid • Impact of patient decision aid is assessed (e.g., through knowledge questionnaires)	• Users are asked their thoughts & opinions of the patient decision aid • Impact of patient decision aid is assessed (e.g., through knowledge questionnaires) • Users are observed interacting with the patient decision aid	• Users are asked their thoughts & opinions of the patient decision aid • Impact of patient decision aid is assessed (e.g., through knowledge questionnaires) • Users are observed interacting with the patient decision aid
Sociodemographic data reported about users	• Age • Sex and/or Gender • Education	• Age • Sex and/or Gender • Education	• Age • Sex and/or Gender • Education
(patients, family members, caregivers, surrogates) involved	• Clinical Characteristics	• Clinical Characteristics • Race and/or Ethnicity	• Clinical Characteristics • Race and/or Ethnicity • Literacy or Health Literacy
Involvement of health professionals who are not members of the research team	• 5 health professionals who are not members of the research team are involved in some way	• 13 health professionals who are not members of the research team are involved in some way • Health professionals are asked their thoughts & opinions of the patient decision aid	• 26 health professionals who are not members of the research team are involved in some way • Health professionals are asked their thoughts & opinions of the patient decision aid • Impact of patient decision aid on clinical practice is assessed (e.g., on shared decision making practices) • Health professionals are observed interacting with the patient decision aid
Advisors	• None	• Expert panel (panel of academics, health professionals, etc.)	• Expert panel (panel of academics, health professionals, etc.) • Formal links with a specific patient or consumer organization • Users are involved in an advisory capacity (as individual advisors, as part of an advisory panel, as members of the research team)

^*We consider these processes applicable to the design and development of new patient decision aids without an existing template and in the absence of urgent needs. For a design and development process to meet the standards of any of the three processes, it should typically include all steps in that column, acknowledging that sometimes, certain steps may not be included for valid reasons (e.g., health professionals may not be involved if the patient decision aid is explicitly intended to be used only by the patient; projects that use in-depth qualitative methodologies rather may have fewer numbers of people involved.)

^**An iterative cycle is defined as, “Your team developed something and showed it to at least one person outside the team before making changes in response to their reactions or feedback. Each new cycle leads to a version of the tool that has been revised in some small or large way.”

^***Here, users are defined as patients, family members, caregivers, or surrogates. While we acknowledge that health professionals may be deeply involved in the use of patient decision aids, ultimately, patient decision aids are designed for the people whose health or family may be affected by the decision. Therefore, for brevity, we refer to users when referencing patients, family members, caregivers, or surrogates, and health professionals when referencing the people who provide health care to users.

^****A formal needs assessment was defined as per the authors’ reports, meaning that authors reported conducting a “needs assessment.” An informal needs assessment was defined as using the methods of needs assessments (e.g., interviewing patients to explore what support they would like to have when making the decision) without naming it as such.

Application of User-Centeredness in Development Studies of Patient Decision Aids

Reviewer Ratings

A total of 66 articles were reviewed by 5 reviewers (3 working independently and 2 working together, see Appendix Table 1.) The mean overall UCD-11 rating was 6.45 (SD = 3.10), with a range from 0 to 11. From the independent reviewers’ ratings, the item least frequently endorsed was item 5: “Were potential end users observed using the tool in any way?” (reviewer ratings indicating yes, n = 19, 29%). Both items related to iterative cycles during the development process were also uncommon. Conversely, the most widely endorsed item was item 1, “were potential end users involved in any steps to help understand users and their needs?” (yes, n = 45, 68%). Table 3 shows a summary of item ratings from both reviewers and authors. We observed no statistically significant difference in the UCD-11 scores for the 12 articles where authors did not provide ratings (mean=5.00, SD=3.38) and the 54 where they did provide ratings (mean=6.78, SD=2.98, P=0.07).

Table 3.

UCD-11 Ratings by Reviewers and Authors for Studies Reporting on the Development of Patient Decision Aids^*

	Articles rated only by reviewers (n = 12)	Articles rated by reviewers and authors
		Reviewer ratings (n=54)	Author ratings (n = 54)
Potential end users
1 Involved in any steps to help understand users and their needs?	9 (75.0%)	36 (66.7%)	51 (94.4%)
2 Involved in designing, developing, and/or refining prototype?	4 (33.3%)	34 (63.0%)	51 (94.4%)
3 Involved in steps intended to evaluate prototypes or a final version?	8 (66.7%)	49 (90.7%)	52 (96.3%)
4 Asked their opinions of the tool in any way?	8 (66.7%)	44 (81.5%)	54 (100%)
5 Observed using the tool in any way?	2 (16.7%)	17 (31.5%)	32 (59.3%)
Development process
6 Have 3 or more iterative cycles?	3 (25.0%)	22 (40.7%)	46 (85.2%)
7 Changes between iterative cycles explicitly reported?	2 (16.7%)	23 (42.6%)	34 (63.0%)
Health professionals
8 Asked their opinion at any point?	8 (66.7%)	43 (79.6%)	53 (98.1%)
9 Consulted before a first prototype?	6 (50.0%)	34 (63.0%)	51 (94.4%)
10 Consulted between initial and final prototypes?	7 (58.3%)	38 (70.4%)	52 (96.3%)
11 Expert panel involved?	3 (25.0%)	26 (48.1%)	41 (77.4%)

^*We did not receive author ratings for 12 articles. 54 authors of the 66 total articles provided data on the UCD-11.

Author Feedback

The response rate from authors who were contacted to complete the UCD-11 instrument was high (54/66, 82%). Overall, the responses indicated a high UCD-11 score (mean = 9.62, SD = 1.16), with a range from 5 to 11. All authors asked the opinions of users about the aids in some way. The UCD-11 item that was least frequently reported was item 5 regarding potential end users being observed using the tool (yes, n = 32, 59%).

Comparing Reviewer and Author Feedback

Reviewers’ UCD-11 ratings (M = 6.45, SD = 3.10) were significantly lower than authors’ ratings (M = 9.62, SD = 1.16; t₅₂ = 6.94, P < .001, 95% CI: 1.94, 3.53). The difference between reviewers’ and authors’ ratings (computed by subtracting the reviewers’ ratings from authors’ ratings) ranged from −3 (indicating a higher reviewer rating than author rating) to 10 (indicating a higher author rating than reviewer). The most frequent scoring difference was 2 (n = 12, 23%), with an average difference of 2.74 (SD = 2.87). Of note, large differences between reviewers’ and authors’ ratings about iterations of prototypes during the development process were observed, suggesting significant under-reporting of these activities. Studies rated the highest on the UCD-11 (i.e., scores of 10 or 11) by reviewers and authors are given in Appendix Table 1.

As shown in Table 4, the number of articles reporting on the development of patient decision aids has consistently increased over time. However, there is no significant difference in the UCD-11 score and year the article was published for ratings from reviewers or authors.

Table 4.

UCD-11 Ratings for Studies Reporting on the Development of Patient Decision Aids by Year of Publication^*

Publication year	Articles	UCD-11 reviewer	UCD-11 author
	N (author n)*	Mean (SD)	Mean (SD)
2014	5 (4)	6.20 (3.96)	10.25 (0.96)
2015	5 (4)	5.80 (2.39)	10.00 (0.82)
2016	14 (12)	6.57 (2.79)	8.92 (1.56)
2017	12 (9)	5.83 (3.59)	9.89 (1.27)
2018	15 (13)	6.93 (3.56)	9.46 (0.88)
2019	15 (11)	6.67 (2.79)	10.00 (0.77)

^*N is total number of articles, and n is number rate by authors.

Reporting Checklist for Developers

The DEVELOPTOOLS Reporting Checklist (Table 5) builds on the UCD-11 measure by incorporating further description beyond yes/no answers to questions, and by adding nine additional items not included in the measure but deemed to be important by the research team based on previous work, including consultation with 15 external experts. These reporting elements are designed to be applicable to the design and development process of patient decision aids as well as other personal health tools and to complement existing reporting guidelines for trials of patient decision aids.(5) The checklist may be used as the basis for manuscripts about the development process, which may be posted as preprints whether or not journal publication is a priority for the team. They may also be included as supplemental files with manuscripts or uploaded with other materials such as questionnaires on platforms that host research project materials (e.g., Open Science Framework, Zenodo).

Table 5.

DEVELOPTOOLS Reporting Checklist.

Item	Explanation	UCD-11 Measure Scoring	Reporting Checklist Additional Question(s)
Factor: Pre-prototype involvement
1. Were potential users (patients, caregivers, family and friends, surrogates) involved in any steps to help understand users (e.g., who they are, in what context might they use the tool) and their needs?	Such steps could include various forms of user research, including formal or informal needs assessment, focus groups, surveys, contextual inquiry, ethnographic observation of existing practices, literature review in which users were involved in appraising and interpreting existing literature, development of user groups, personas, user profiles, tasks, or scenarios, or other activities.	Yes = 1, No = 0	If yes, what did you do (e.g., interviews, focus groups, surveys)? How many users of each type were involved in each of these steps?
2. Were potential users (patients, caregivers, family and friends, surrogates) involved in any steps of designing, developing, and/or refining a prototype?	Such steps could include storyboarding, reviewing the draft design or content prior to starting to develop the tool, and designing, developing, or refining a prototype.**	Yes = 1, No = 0	If yes, what did you do (e.g., co-design workshops)? How many users of each type were involved in each of these steps?
Factor: Iterative responsiveness
3. Were potential users (patients, caregivers, family and friends, surrogates) involved in any steps intended to evaluate prototypes of the tool or a final version of the tool?	Such steps could include feasibility testing, usability testing with iterative prototypes, pilot testing, a randomized controlled trial of a final version of the tool, or other activities.	Yes = 1, No = 0	If yes, what did you do? How many users of each type were involved in each of these steps?
4. Were potential users (patients, caregivers, family and friends, surrogates) asked their opinions of prototypes of the tool or a final version of the tool in any way?	For example, they might be asked to voice their opinions in a focus group, interview, survey, or through other methods.	Yes = 1, No = 0	If yes, what did you do? How many users of each type were involved?
5. Were potential users (patients, caregivers, family and friends, surrogates) observed using the tool in any way?	For example, they might be observed in a think-aloud study, cognitive interviews, through passive observation, logfiles, or other methods.	Yes = 1, No = 0	If yes, what did you do? How many users of each type were involved?
6. Did the development process have three or more iterative cycles?	The definition of a cycle is that your team developed something and showed it to at least one person outside the team before making changes in response to their reactions or feedback. Each new cycle leads to a version of the tool that has been revised in some small or large way.	Yes = 1, No = 0	If yes or no, how many cycles did you have?
7. Were changes between iterative cycles explicitly reported in any way?	For example, the team might have explicitly reported them in a peer-reviewed paper or in a technical report. In the case of rapid prototyping, such reporting could be, for example, a list of design decisions made and the rationale for the decisions.	Yes = 1, No = 0	If yes, what did you do?
Factor: Other expert involvement
8. Were health professionals asked their opinion of the tool at any point?	Health professionals could be any relevant professionals, including physicians, nurses, allied health professionals, etc. These professionals are not members of the research team. They provide care to people who are likely users of the tool. Asking their opinion means simply asking for feedback, in contrast to, for example, observing their interaction with the tool or assessing the impact of the tool on health professionals’ behavior.	Yes = 1, No = 0	If yes, what did you do?
9. Were health professionals consulted before a first prototype was developed?	Consulting before a first prototype means consulting prior to developing anything. This may include a variety of consultation methods.	Yes = 1, No = 0	If yes, what did you do?
10. Were health professionals consulted between initial and final prototypes?	Consulting between initial and final prototypes means some initial design of the tool was already created when consulting with health professionals.	Yes = 1, No = 0	If yes, what did you do?
11. Was an expert panel involved?	An expert panel is typically an advisory panel composed of experts in areas relevant to the tool if such experts are not already present on the research team; for example, plain language experts, accessibility experts, designers, engineers, industrial designers, digital security experts, etc. These experts may be health professionals, but not health professionals who would provide direct care to end users.	Yes = 1, No = 0	If yes, who was involved?
Additional elements in DEVELOPTOOLS Reporting Checklist
12. Was a formal advisory panel of users involved?	Such formal panels could be existing panels or they could be assembled for the project.	n/a	If yes, what kind of panel was it and how was the panel assembled and involved?
13. Were users (patients, caregivers, family and friends, surrogates), health professionals, and other relevant stakeholders involved as members of the research team?	User involvement on the research team implies that users had some level of decisional authority in the research plan. Similarly, health professional involvement implies that health professionals had some level of decisional authority in the research plan.	n/a	If yes, who were the users, health professionals, and other relevant stakeholders? What perspectives did they bring, and how were they involved?
14. Were members of populations marginalized by social norms and policies involved?	Populations who have been marginalized by social norms and policies are social groups with a higher risk of health problems.(24) These groups include but are not limited to: people who are poor, discriminated against, stigmatized, marginalized or disenfranchised due to psychological, physical, sensory or cognitive characteristics (e.g., mental illness, low literacy, disability), socio-economic or socio-cultural characteristics (e.g., education, income, race/ethnicity, language, gender identity, sexual orientation, immigration status) or for other reasons (e.g., alcohol or drug dependencies).	n/a	If yes, what populations were involved, how were they recruited, and how were they involved?
15. How many users (patients, caregivers, family and friends, surrogates) and health professionals were involved in total, and of each type?	People should be reported according to group.	n/a	How many people of each type were involved?
16. Does the tool have a defined purpose?	The tool’s purpose may be to support shared decision making, to enable a person to accomplish a physical or cognitive task, to support self-management, or other purposes.	n/a	What is the purpose of the tool?
17. Is the tool intended to be used in a particular context?	Tools may be intended to be used at home at any time, in a clinic during a consultation, or in other place/time contexts.	n/a	In what context is the tool intended to be used?
18. Were any methods used to facilitate sharing of perspectives between groups?	For example, workshops involving users (patients, families, caregivers, surrogates), health professionals, researchers, and other stakeholders may be used for this purpose.	n/a	If yes, what methods were used?
19. Were users (patients, caregivers, family and friends, surrogates) involved from the outset of the project?	Users may be involved from the very beginning of a project to, among other things, help establish the purpose of the patient decision aid, its audience, and the scope of its content.	n/a	At what point were users involved?
20. Were translation and cultural adaptation used to render the patient decision aid available to users across languages and cultures?	For example, a patient decision aid might be developed in one language, then translated into one or more other languages and culturally adapted and validated to ensure it is acceptable to members of other cultures.(23)***	n/a	If yes, what was done?

^*Activities like think aloud and other evaluative exercises are considered prototype development activities (item 2) if they occur during rapid cycles of development (e.g., a co-design workshop) and if the users are involved in interpreting the data. If, on the other hand, users are simply shown the prototype and asked to think aloud or asked their opinions, this is considered a step intended to evaluate the tool (item 3.)

^**Translation of a decision aid from one language (e.g., English) to another without careful consideration of cultural factors and needed adaptations is strongly discouraged.

Discussion

We conducted an update of the systematic development process recommended by the IPDAS Collaboration to guide the development of patient decision aids. In this update we found strong evidence of involvement of users and health professionals in development processes, although the degree of engagement was variable. We further found that authors consistently under-reported user and health professional involvement in publications about their decision aids across all phases of development. We identified a subset of 8 articles that were rated highest (i.e., those scoring highest on the UCD-11) by the reviewers and scored highest by the authors to serve as potential exemplars for other developers who wish to maximally involve users and health professionals. Of note, these exemplar papers focused primarily on details of the development process rather than reporting results of a trial of decision aids outcomes, and many included as a figure an adapted version of the development model from Coulter et al.(1)

To support teams in developing new patient decision aids, we describe a spectrum of user-centeredness in understanding users’ and health professionals’ needs, prototype development, usability testing, evaluation of decision aids, attention to patient subgroups, and involvement of health experts and advisors. A minimal process, which characterizes development steps included in the large majority of decision aid development projects, relies on literature reviews to identify users’ and health professionals’ needs and context, a few iterative cycles, some pilot or usability testing with users and health professionals, and limited attention to subgroups of patients. In practice, teams must make decisions about the strategies they will employ to develop decision aids based on considerations related to time, costs, and urgency of the topic. It is also important to note that the development process is ongoing and should continue as the decision aid is used in real world decision making and as new data arise about new decision-making needs, adaptations for additional user groups, and so on. Ongoing development is challenging for decision aids created through research projects, although funders are increasing requiring dissemination plans that can include the involvement of professional organizations. Support for ongoing development outside of research contexts is needed, such as through organizations with an interest in the aid and willingness to take ownership of its continued revision. To support implementation and continued use of patient decision aids, researchers may wish to involve stakeholders from such organizations along with users and health professionals in an integrated knowledge translation approach.(11, 12)

It is not the purpose of this update to propose rigid, overly prescriptive, complex and time-consuming standards for engaging users, health professionals, and other stakeholders in the design of decision aids. This concern is even more important for aids developed outside of research studies, which likely constitutes the majority of decision aids (see The Ottawa Hospital A to Z Inventory of Decision Aids, for example, https://decisionaid.ohri.ca/AZinvent.php). For real world implementation of patient decision aids and shared decision making at scale, some developers may use templates, standardized processes, or common resources for producing new decision aids.(13, 14) Such templates may be initially developed and tested thoroughly in a similar way as the maximum process presented above. Then, when the template is used as a starting point for developing new patient decision aids for a specific clinical situation, it may be only necessary to use a minimal process or similarly abridged process. The Center for Shared Decision Making in Denmark has developed a generic template as a platform from which new patient decision aids that can be developed and adapted to different specific decisions.(15–17) The generic patient decision aid was developed through a variety of methods such as multidisciplinary steering group meetings, interviews with users, health professionals and other stakeholders, observations of consultations and tests for getting feedback from all stakeholders and more than 20 iterative processes before the patient decision aid template was finalized. The template contains fixed text which is “pre-printed” and which cannot be changed (to ensure that the tool complies with IPDAS) as well as empty space for text/statistics/icons to be filled in for the specific situation. A manual accompanies the template on how to develop your own decision aid and guidelines for the development process and minimum testing of the specific decision aid. Similar examples of well-established, well-researched approaches to developing decision aids that have been applied successfully to numerous decision contexts include the Ottawa Decision Support Framework, (18), the Making GRADE the Irresistible Choice MAGICapp (https://magicevidence.org/), Decision Boxes (19, 20) and Option Grids. (21)

For patient decision aids developed outside the structure of existing templates, the development process need not be onerous. Patients who have experienced the decision and outcomes related to choices can be interviewed to explore their needs and context, and participate in testing prototype versions of the aid. Members of the clinical team can help select formats for aids, when aids should be delivered, what content might be addressed based on their experiences with patients, and in general provide input that will enhance the feasibility of using the aids in clinical care. The same health professionals who champion the use of patient decision aids can serve on development teams.

There are situations when a more rapid development process will be needed to respond to an urgent need. Stacey and colleagues (22) describe the rapid development of two decision aids in response to the COVID-19 pandemic. They relied on an expedited Ottawa Decision Support Framework patient decision aid process and the Ottawa Framework template in producing and disseminating these two aids. Several aspects of their development approach should be highlighted here. First, the team deemed it infeasible to have patients serve on the development team, although health professionals were represented. Second, needs assessments were limited to existing, available information (e.g., media responses from the public, steering group views) and no new data was collected. Third, evidence about the outcome probabilities was limited and changing rapidly, and could not be included in the aid. Fourth, user testing was limited to use of the aids by members of the steering committee. Finally, effectiveness of the Ottawa template had been previously demonstrated in randomized trials and was not re-assessed for these aids. Key conclusions from Stacey et al., are that rapid development processes are feasible for time-sensitive decision aids, but should be done by experienced developers with an existing, well-established template or approach to development.(22)

Attention to the needs of disadvantaged groups in providing decision support is an ongoing challenge.(23) When producing decision aids, developers use a variety of strategies to involve members of groups that are marginalized by society or by policy. Dugas and colleagues (24) found that developers who specifically included members of such groups relied more on informal needs assessment and community-based organizations than developers who did not specifically involve vulnerable groups, and made a practice of going out to the groups (e.g., holding meetings at community locations) rather than expecting members of groups to come to them. They further noted several barriers to involving persons from vulnerable groups, including logistical issues (e.g., transportation costs), trust, and additional project resources needed to facilitate participation. Facilitators to participation include flexibility in scheduling and location, adapting technology, taking the time to build trusting relationships, and providing incentives.

The DEVELOPTOOLS reporting checklist offers a standardized approach to more fully describe the design and development process for patient decisions aids. It can be appended to descriptions of new patient decision aids posted online and also used to complement the current SUNDAE reporting guidelines for studies that evaluate patient decision aids.(5) Part of the challenge in providing a more robust description of development processes is that journals place limitations on the length of articles. When reporting the results of comparative trials of decision aids, for example, attention appropriately needs to be given to the methods of the trial rather than details about development of the aids being tested. We encourage developers to publish their development processes and describe templates they used. We further encourage developers who elect not to publish separate papers about the development process to provide supplementary material about their development processes, including brief responses to the elements of the DEVELOPTOOLS reporting checklist so adopters of decision aids have a complete picture of how an aid was produced.

Limitations

This update has several limitations. First, evidence about how development processes of varying complexity relate to patient outcomes or improved decision-making processes is lacking. For example, evidence is lacking about whether a more involved development process leads to greater implementation success compared to a minimal process. Second, we do not report on what is likely the large number of patient decision aids produced outside of research projects, which are never published in peer-reviewed journals, and may lack involvement of experienced researchers. Finally, not all authors responded to our query to rate their development processes with the UCD-11.

Conclusions

Developers of patient decision aids have embraced principles of user-centered design. The phased model published by Coulter and colleagues in 2013 has made the process of developing patient decision aids accessible to many new developers. Developers may wish to further embrace iterative methods of development. The effectiveness of patient decision aids in improving the decision-making process between patients and health professionals, and improving the quality and outcomes of health care decisions by patients, is well-established.(25) Implementation of patient decision aids in real world health care remains an essential challenge for the field.(2) User and health professional involvement in the development process is likely a key to achieving this goal.

Highlights.

• Users of patient decision aids deserve assurance that the processes used to create these tools follow acceptable standards.

• Since the last update of the IPDAS standards, principles of user-centered design have been widely adopted by developers.

• Developers consistently under-report the involvement of users when describing the development of decision aids.

• Templates and a checklist may help developers more fully report on the processes they use in creating these tools.

Appendix Figure 1.

Overview of Search and Review Strategy.

Appendix Table 1.

Articles Rated for User-Centeredness of Patient Decision Aid Design Process with the UCD-11.

Study	Clinical Context
Bansback 2019 (26)	Relapsing Multiple Sclerosis
Bansback 2019 (27)	Knee Arthroplasty
Barr 2019 (28)	Depression
Grimmett 2019 (29)	Breast Cancer Genetic Testing
Hahlweg 2019 (30)	Breast Cancer
Jackson 2019 (31)	Ulcerative Colitis
Loewen 2019 (32)	Atrial Fibrillation Stroke Prevention Therapy
Moore 2019 (33)	Adolescent Obesity
Peresterlo-Perez 2019 (34)	Colorectal Cancer Screening
Reumkens 2019 (35)	Reproductive Choices, Genetic Predisposition to Cancer
Scalia 2019 (36)	Prostate Cancer Screening
Shahrzad 2019 (37)	Breast Cancer Patients
Squires 2019 (38)	Contralateral Prophylactic Mastectomy (Breast Cancer)
van den Berg 2019 (39)	Postoperative Analgesia
Wood 2019 (40)	Cervical Cancer Screening Preferences
Ager 2018* (41)	Contralateral Prophylactic Mastectomy (Breast Cancer)
Fowler 2018 (42)	Breast Cancer Screening Decisions (women with dementia)
Gabel 2018* (43)	Colon Cancer Screening
Harmsen 2018* (44)	Risk-Reducing Surgery (BRCA1/2)
Hooiveld 2018 (45)	Contraceptives
Klaassen 2018 (46)	Curatively Treated Breast Cancer Surgery Aftercare
Lewis 2018* (47)	Implantable Cardioverter-Defibrillator Replacement
McDonnell 2018 (48)	Lung Cancer Screening
Probst 2018 (49)	Syncope
Quigley 2018 (50)	Dysvascular Partial Foot and Transtibial Amputations
Robertson 2018 (51)	Pediatric Oncology Clinical Trial Enrollment
Santerre-Theil* 2018 (52)	Share Genetic Results with Underage Children (BRCA1/2)
Scalia 2018 (53)	Prostate-Specific Antigen Test
Sowan 2018 (54)	Peripherally Inserted Central Venous Catheter Procedure
Willis 2018 (55)	Clinically Actionable Research Findings (Genomic Research Participants)
Coxeter 2017 (56)	Antibiotic Use for Common Acute Respiratory Infections
Etnel 2017 (57)	Congenital Heart Disease
Gagne 2017 (58)	Adults with Asthma
Jones 2017 (59)	Fertility Preservation
Malloy 2017 (60)	Initiation of Antipsychotic Medications (Persons with Dementia)
Moore 2017 (61)	Prematurity in Gestational Age (Parents)
Nota 2017* (62)	Disease Modifying Anti-Rheumatic Drugs
Perestelo-Perez 2017 (63)	Depression
Sajeev 2017 (64)	Nutrition Support in Pediatric Oncology
Savelberg 2017 (65)	Early Stage Breast Cancer
Thompson 2017 (66)	Acid Reflux Medication
Toledo-Cavarri 2017 (67)	Breast Cancer Screening
Al-Itejawi 2016 (68)	Prostate Cancer
Beach 2016 (69)	Anemia Treatment (Kidney Disease)
Beaulac 2016 (70)	Depression
Birch 2016 (71)	Genome-Wide Sequencing
Durand 2016 (72)	Breast Cancer
Hiligsmann 2016 (73)	Osteoporosis (Post-Menopausal Women)
Maguire 2016* (74)	Adenotonsillectomy (Patient Decision Aid for Parents of Children with Sleep Disordered Breathing)
Marshman 2016 (75)	Fixed Orthodontic Appliances
Patzer 2016 (76)	Kidney Transplant vs. Dialysis
Perestelo-Perez 2016 (77)	Type 2 Diabetes
Stacey 2016 (78)	Cochrane Review Translation to Decision Making
Trenaman 2016 (79)	Obstructive Sleep Apnea
Winterbottom 2016 (80)	Dialysis
Zdenkowski 2016 (81)	Neoadjuvant Systemic Therapy (Breast Cancer)
Feenstra 2015 (82)	Type 1 Diabetes (Involving Children in Decisions)
Kaiser 2015 (83)	Oral Anticoagulant Treatment (Atrial Fibrillation)
Shillington 2015 (84)	Type 2 Diabetes
Shorten 2015 (85)	Mode of Delivery after Caesarean Section
Thompson 2015 (86)	Destination Therapy Left Ventricular Assist Device (Advanced Heart Failure)
Barton 2014* (87)	Rheumatoid Arthritis
Lau 2014 (88)	Lung Cancer Screening
Ng 2014 (89)	Type 2 Diabetes
Schoorel 2014 (90)	Mode of Delivery after Caesarean Section
Volk 2014 (91)	Lung Cancer Screening

^*Denotes studies rated highest on the UCD-11 by reviewers and authors.