Table 5.
DEVELOPTOOLS Reporting Checklist.
| Item | Explanation | UCD-11 Measure Scoring | Reporting Checklist Additional Question(s) |
|---|---|---|---|
| Factor: Pre-prototype involvement | |||
| 1. Were potential users (patients, caregivers, family and friends, surrogates) involved in any steps to help understand users (e.g., who they are, in what context might they use the tool) and their needs? | Such steps could include various forms of user research, including formal or informal needs assessment, focus groups, surveys, contextual inquiry, ethnographic observation of existing practices, literature review in which users were involved in appraising and interpreting existing literature, development of user groups, personas, user profiles, tasks, or scenarios, or other activities. | Yes = 1, No = 0 | If yes, what did you do (e.g., interviews, focus groups, surveys)? How many users of each type were involved in each of these steps? |
| 2. Were potential users (patients, caregivers, family and friends, surrogates) involved in any steps of designing, developing, and/or refining a prototype? | Such steps could include storyboarding, reviewing the draft design or content prior to starting to develop the tool, and designing, developing, or refining a prototype.** | Yes = 1, No = 0 | If yes, what did you do (e.g., co-design workshops)? How many users of each type were involved in each of these steps? |
| Factor: Iterative responsiveness | |||
| 3. Were potential users (patients, caregivers, family and friends, surrogates) involved in any steps intended to evaluate prototypes of the tool or a final version of the tool? | Such steps could include feasibility testing, usability testing with iterative prototypes, pilot testing, a randomized controlled trial of a final version of the tool, or other activities. | Yes = 1, No = 0 | If yes, what did you do? How many users of each type were involved in each of these steps? |
| 4. Were potential users (patients, caregivers, family and friends, surrogates) asked their opinions of prototypes of the tool or a final version of the tool in any way? | For example, they might be asked to voice their opinions in a focus group, interview, survey, or through other methods. | Yes = 1, No = 0 | If yes, what did you do? How many users of each type were involved? |
| 5. Were potential users (patients, caregivers, family and friends, surrogates) observed using the tool in any way? | For example, they might be observed in a think-aloud study, cognitive interviews, through passive observation, logfiles, or other methods. | Yes = 1, No = 0 | If yes, what did you do? How many users of each type were involved? |
| 6. Did the development process have three or more iterative cycles? | The definition of a cycle is that your team developed something and showed it to at least one person outside the team before making changes in response to their reactions or feedback. Each new cycle leads to a version of the tool that has been revised in some small or large way. | Yes = 1, No = 0 | If yes or no, how many cycles did you have? |
| 7. Were changes between iterative cycles explicitly reported in any way? | For example, the team might have explicitly reported them in a peer-reviewed paper or in a technical report. In the case of rapid prototyping, such reporting could be, for example, a list of design decisions made and the rationale for the decisions. | Yes = 1, No = 0 | If yes, what did you do? |
| Factor: Other expert involvement | |||
| 8. Were health professionals asked their opinion of the tool at any point? | Health professionals could be any relevant professionals, including physicians, nurses, allied health professionals, etc. These professionals are not members of the research team. They provide care to people who are likely users of the tool. Asking their opinion means simply asking for feedback, in contrast to, for example, observing their interaction with the tool or assessing the impact of the tool on health professionals’ behavior. | Yes = 1, No = 0 | If yes, what did you do? |
| 9. Were health professionals consulted before a first prototype was developed? | Consulting before a first prototype means consulting prior to developing anything. This may include a variety of consultation methods. | Yes = 1, No = 0 | If yes, what did you do? |
| 10. Were health professionals consulted between initial and final prototypes? |
Consulting between initial and final prototypes means some initial design of the tool was already created when consulting with health professionals. | Yes = 1, No = 0 | If yes, what did you do? |
| 11. Was an expert panel involved? | An expert panel is typically an advisory panel composed of experts in areas relevant to the tool if such experts are not already present on the research team; for example, plain language experts, accessibility experts, designers, engineers, industrial designers, digital security experts, etc. These experts may be health professionals, but not health professionals who would provide direct care to end users. | Yes = 1, No = 0 | If yes, who was involved? |
| Additional elements in DEVELOPTOOLS Reporting Checklist | |||
| 12. Was a formal advisory panel of users involved? | Such formal panels could be existing panels or they could be assembled for the project. | n/a | If yes, what kind of panel was it and how was the panel assembled and involved? |
| 13. Were users (patients, caregivers, family and friends, surrogates), health professionals, and other relevant stakeholders involved as members of the research team? |
User involvement on the research team implies that users had some level of decisional authority in the research plan. Similarly, health professional involvement implies that health professionals had some level of decisional authority in the research plan. | n/a | If yes, who were the users, health professionals, and other relevant stakeholders? What perspectives did they bring, and how were they involved? |
| 14. Were members of populations marginalized by social norms and policies involved? | Populations who have been marginalized by social norms and policies are social groups with a higher risk of health problems.(24) These groups include but are not limited to: people who are poor, discriminated against, stigmatized, marginalized or disenfranchised due to psychological, physical, sensory or cognitive characteristics (e.g., mental illness, low literacy, disability), socio-economic or socio-cultural characteristics (e.g., education, income, race/ethnicity, language, gender identity, sexual orientation, immigration status) or for other reasons (e.g., alcohol or drug dependencies). | n/a | If yes, what populations were involved, how were they recruited, and how were they involved? |
| 15. How many users (patients, caregivers, family and friends, surrogates) and health professionals were involved in total, and of each type? | People should be reported according to group. | n/a | How many people of each type were involved? |
| 16. Does the tool have a defined purpose? | The tool’s purpose may be to support shared decision making, to enable a person to accomplish a physical or cognitive task, to support self-management, or other purposes. | n/a | What is the purpose of the tool? |
| 17. Is the tool intended to be used in a particular context? | Tools may be intended to be used at home at any time, in a clinic during a consultation, or in other place/time contexts. | n/a | In what context is the tool intended to be used? |
| 18. Were any methods used to facilitate sharing of perspectives between groups? | For example, workshops involving users (patients, families, caregivers, surrogates), health professionals, researchers, and other stakeholders may be used for this purpose. | n/a | If yes, what methods were used? |
| 19. Were users (patients, caregivers, family and friends, surrogates) involved from the outset of the project? | Users may be involved from the very beginning of a project to, among other things, help establish the purpose of the patient decision aid, its audience, and the scope of its content. | n/a | At what point were users involved? |
| 20. Were translation and cultural adaptation used to render the patient decision aid available to users across languages and cultures? | For example, a patient decision aid might be developed in one language, then translated into one or more other languages and culturally adapted and validated to ensure it is acceptable to members of other cultures.(23)*** | n/a | If yes, what was done? |
*
Activities like think aloud and other evaluative exercises are considered prototype development activities (item 2) if they occur during rapid cycles of development (e.g., a co-design workshop) and if the users are involved in interpreting the data. If, on the other hand, users are simply shown the prototype and asked to think aloud or asked their opinions, this is considered a step intended to evaluate the tool (item 3.)
**
Translation of a decision aid from one language (e.g., English) to another without careful consideration of cultural factors and needed adaptations is strongly discouraged.