Table 4.
Agreement on the items for usability evaluation involving users (N=27).
| Planning and reporting procedures for usability evaluation with users | Items grouped from 1 to 3, n (%) | Items grouped from 4 to 6, n (%) | Items grouped from 7 to 9, n (%) | ||||
| Usability assessment moderator | |||||||
|
|
Determine the number of usability assessment moderators. | 1 (4) | 13 (48) | 13 (48) | |||
|
|
Provide the rationale used to establish the number of usability assessment moderators. | 2 (7) | 10 (37) | 15 (56) | |||
|
|
Specify as inclusion criteria having previous experience with usability evaluation with users or consider adequate training and provide details of the training plan. | 0 (0) | 2 (7) | 25 (93) a | |||
|
|
Detail inclusion and exclusion criteria other than previous experience (eg, academic background or age). | 0 (0) | 9 (33) | 18 (67) | |||
|
|
Specify whether the usability assessment moderators are external to the service or product development team. | 0 (0) | 4 (15) | 13 (85) | |||
|
|
Specify if observers are included, define their responsibilities, and collect their characteristics (eg, gender, academic background, and previous experience in usability evaluation).b | 0 (0) | 7 (26) | 20 (74) | |||
|
|
Detail the usability assessment moderators’ characteristics that should be collected (eg, gender, academic background, and previous experience conducting usability evaluation).c | 0 (0) | 9 (33) | 18 (67) | |||
| Participants | |||||||
|
|
Determine sample size (ie, the total number of participants involved in the evaluation). | 0 | 3 (11) | 24 (89) | |||
|
|
Provide a rationale to establish the sample size. | 0 | 6 (22) | 21 (78) | |||
|
|
Provide clear inclusion and exclusion criteria (eg, profile definition including age, gender, educational level, digital literacy, and previous experience using the product or service being evaluated). | 0 | 1 (4) | 26 (96) | |||
|
|
Provide sampling methods (eg, random, systematic, cluster, convenience, and snowball). | 0 | 8 (30) | 19 (70) | |||
|
|
Indicate the setting of participants’ recruitment (eg, community and hospital). | 0 | 4 (15) | 23 (85) | |||
|
|
Detail clinical conditions (if relevant for the study) (eg, asymptomatic or with a specific clinical condition or from a specific group—occupational group, the severity of the clinical condition, disabilities, cognitive impairment). | 0 | 2 (7) | 25 (93) | |||
|
|
Detail the participant’s characteristics that should be collected (such as age, gender, educational level, and digital literacy).c | 0 | 0 | 27 (100) | |||
| Usability evaluation method and usability evaluation technique | |||||||
|
|
Specify whether a combination of usability evaluation methods was used (eg, using both inquiry and test methods). | 2 (7) | 4 (15) | 21 (78) | |||
|
|
Specify whether a combination of usability evaluation techniques was used (eg, for the inquiry method, combine the questionnaire and interview techniques). | 1 (4) | 4 (15) | 22 (82) | |||
|
|
Provide the rationale for the choice of usability evaluation methods and techniques. | 0 | 5 (19) | 22 (82) | |||
|
|
Describe the usability evaluation methods and techniques used and how they are implemented.b | 0 | 2 (7) | 25 (93) | |||
|
|
When using measuring instruments such as scales or questionnaires, give indicators of their validity and reliability. | 0 | 2 (7) | 25 (93) | |||
|
|
Describe the data analysis plan for both quantitative and qualitative data.c | 0 | 4 (15) | 23 (85) | |||
| Tasks | |||||||
|
|
Provide a detailed description of tasks or present the session script. | 0 | 1 (4) | 26 (96) | |||
|
|
Indicate the total number of tasks. | 0 | 6 (22) | 21 (78) | |||
|
|
Detail the task-related outcomes and how they are measured (eg, task completion and duration and number of errors).c | 0 | 5 (19) | 22 (82) | |||
|
|
Detail the conditions for carrying out the tasks (eg, with or without supervision, individually or in the group, with or without a period for familiarization with the digital product or service).c | 0 | 0 | 27 (100) | |||
|
|
Detail the instructions to participants and the way they are presented (eg, verbally, written, and both) and registered (eg, audio, video, screen recorder, and notes from an observer).c | 0 | 2 (7) | 27 (100) | |||
| Usability evaluation environment | |||||||
|
|
Justify the choice of the usability evaluation environment (eg, lab or field test and remote or face-to-face test). | 0 | 8 (30) | 19 (70) | |||
|
|
Specify usability evaluation environment requirements (eg, recording equipment or observer room availability). | 0 | 5 (19) | 22 (82) | |||
|
|
Detail the procedures to make the usability evaluation environment safe and comfortable for the participants.c | 0 | 9 (33) | 18 (67) | |||
aValues in italics denote the items that reached consensus on inclusion.
bItems that were rephrased.
cNew items that emerged from round 1.