Table 3.
Results from rounds 2 and 3 of the Delphi panel.
| Questions | Second round score, n (%) | Third round score, n (%) | Consensus | ||||
| 1-3 | 4-6 | 7-9 | 1-3 | 4-6 | 7-9 | ||
| Did the study use valid measurement instruments of usability (ie, there is evidence that the instruments used assess usability)? | 0 (0) | 0 (0) | 23 (100) | 0 (0) | 0 (0) | 23 (100) | Yes |
| Did the study use reliable measurement instruments of usability (ie, there is evidence that the instruments used have similar results in repeated measurements in similar circumstances)? | 0 (0) | 3 (13) | 20 (87) | 0 (0) | 0 (0) | 23 (100) | Yes |
| Was there coherence between the procedures used to assess usability (eg, instruments, context) and study aims? | 0 (0) | 4 (17) | 19 (83) | 0 (0) | 2 (9) | 21 (91) | Yes |
| Did the study use procedures of assessment for usability that were adequate to the development stage of the product/service? | 0 (0) | 5 (22) | 18 (78) | 0 (0) | 4 (17) | 19 (83) | Yes |
| Did the study use procedures of assessment for usability adequate to study participants’ characteristics (eg, children and elderly require different instruments)? | 0 (0) | 2 (9) | 21 (91) | 0 (0) | 1 (4) | 22 (96) | Yes |
| Did the study employ triangulation of methods for the assessment of usability? | 0 (0) | 5 (22) | 18 (78) | 0 (0) | 6 (26) | 17 (74) | Yes |
| Was usability assessed using both potential users and experts? | 0 (0) | 5 (22) | 18 (78) | 0 (0) | 4 (17) | 19 (83) | Yes |
| Were participants who assessed the product/service usability representative of the experts’ population and/or of the potential users’ population? | 0 (0) | 3 (13) | 20 (87) | 0 (0) | 2 (9) | 21 (91) | Yes |
| Was the investigator that conducted usability assessments adequately trained? | 1 (4) | 4 (17) | 18 (78) | 1 (4) | 1 (4) | 21 (91) | Yes |
| Was the investigator that conducted usability assessments external to the process of product/service development? | 1 (4) | 10 (44) | 12 (52) | 1 (4) | 10 (43) | 12 (52) | Noa |
| Was the usability assessment conducted in the real context or close to the real context where product/service is going to be used? | 0 (0) | 5 (22) | 18 (78) | 0 (0) | 2 (9) | 21 (91) | Yes |
| Was the number of participants used to assess usability adequate (whether potential users or experts)? | 0 (0) | 2 (9) | 21 (91) | 0 (0) | 0 (0) | 23 (100) | Yes |
| Were the tasks that serve as the base for the usability assessment representative of the functionalities of the product/service? | 0 (0) | 0 (0) | 23 (100) | 0 (0) | 0 (0) | 23 (100) | Yes |
| Was the usability assessment based on continuous and prolonged use of the product/service over time? | 0 (0) | 9 (39) | 14 (61) | 0 (0) | 6 (26) | 17 (74) | Yes |
| Was the type of analysis adequate to the study’s aims and variables assessed? | 0 (0) | 1 (4) | 22 (96) | 0 (0) | 0 (0) | 23 (100) | Yes |
aThis item was included because most of the experts (n=18, 78%) classified it with 6 or more out of a maximum score of 9.