Table 3.
Study characteristics, including study design, number of patients included, duration of follow-up, method of usability analysis, usability domain, and selected usability results.
| Study | Study design | Number of patients | Duration follow-up | Method of analysis of usability /outcome measure | Aspects of usability measured | Selected quantitative measure of usability |
| Timmers et al [28] | Multicenter RCTa | 213 | 4 weeks | Measurement of patient usage; Interview of small group of patients (n=6) |
Usefulness | App used 26 times/patient; Videos watched 36 times/patient; Qualitative reporting of usefulness |
| Yadav et al [29] | Prospective study (no control) | 107 | 6 months | Self-created questionnaire | Satisfaction; Usefulness |
1% unsatisfied across the questionnaire; 53% very satisfied with effectiveness; 78% very satisfied with app overall; Comfortable: 78% very satisfied; Convenience: 86%-91% very satisfied |
| Ramkumar et al [30] | Prospective study (no control) | 22 | 3 months | Semi-structured interview | Satisfaction; Usefulness |
A1: average score 2.6/10 (1=easiest to use; 10=most difficult) |
| Argent et al [31] | Mixed methods, including prospective study | 15 | 2 weeks | Questionnaires (SUSb and uMARSc); Semi-structured interview |
Satisfaction; System information arrangement; Usefulness |
uMARS average score 4.1/5 (SD 0.39); SUS average score 90.8 (SD 7.8) |
| Brunner et al [32] | Prospective preintervention and postintervention study | 38 | 4 days | Self-created questionnaires; Measurement of usage |
Satisfaction; Usefulness |
66% used the app; 60% satisfied with the app; 85% felt it was helpful |
| van der Meij et al [33] | RCT | 344 | 3 months | Measurement of usage; Self-created questionnaire; Semistructured interviews |
Satisfaction | 49.6% had used the app; Mean score for app 7.6/10 |
| Felbaum et al [34] | Prospective study (no control) | 56 | —d | Self-created questionnaire | Usefulness | Usefulness ranged from 8.39-9.0 out of 10 (Likert scale) |
| Goz et al [35] | Prospective study (no control) | 21 | 2 weeks | Measurement of usage/engagement; Self-created questionnaire |
Satisfaction; Usefulness |
82% satisfied (would recommend to others); 75% found useful (felt the app made it less likely for them to call the clinic); Engagement: 3.38 messages/person over 2 weeks |
| Gunter et al [36] | Prospective study (no control) | 40 | 2 weeks | SUS (questionnaire); Measurement of usage |
Satisfaction; System information arrangement |
SUS average score of 87.2 |
| Gustavell et al [37] | Prospective study (no control) | 6 | 4 weeks | Measurement of usage; Semistructured interviews |
Satisfaction; System information arrangement; Usefulness |
Adherence to reporting daily was 84%; Other measurements qualitative |
| Harder et al [38] | Prospective study (no control) | 4 | 8 weeks | Measurement of usage; Self-created questionnaire |
Satisfaction; System information arrangement; Usefulness |
Overall rating (Likert scale) 4.6/5; All used the app almost daily or several times/day |
| Higgins et al [39] | Retrospective case series | 32 | 6 weeks | Interview; Self-created questionnaire |
Satisfaction | Overall satisfaction was reported as excellent (43%), good (40%), fair (10%), poor (7%); 94% would use the app again |
| Highland et al [40] | RCT | 24 (only 12 assessed usability) | 10 days | SUS questionnaire; Additional questionnaire |
Satisfaction; System information arrangement; Usefulness |
SUS average score 76.26/100; No difference in convenience between intervention and standard of care (telephone follow-up) |
| Khanwalkar et al [41] | Prospective study (no control) | 249 | 3 months | Measurement of usage | None | 77.4% response rate (usage) |
| Mata et al [42] | RCT | 50 | 4 weeks; Satisfaction measured at discharge |
Measurement of usage; Self-created questionnaire using 4 items from S-CAHPSe |
Satisfaction | Usage: postoperative day 0=94%, day 1=82%, day 2=72%, day 3=48%; 4/5 satisfaction across all 4 questions |
| Nilsson et al [43] | Prospective study (no control) | 494 | 14 days | Measurement of usage (response rate) | None | Usage: day 1=86.8%, day 7=69%, day 14=57.5% |
| Pecorelli et al [44] | Prospective study (no control) | 45 | 4 weeks | SUS questionnaire | Satisfaction; System information arrangement |
SUS average score 87/100 |
| Sousa and Turrini [45] | Prospective study (no control) | 30 | — | SUS questionnaire; Satisfaction measured according to experience sampling method technique; Usage |
Satisfaction; System information arrangement |
SUS average score 79.8/100, 73.3% >68 (cutoff), 100% >50 (acceptable); Satisfaction 82.9%; Usage: 100% used at least once, 40% used 2-3 times, 10% used 5 times, 20% used >5 times |
| Sun et al [46] | Prospective study (no control) | 29 | — | CSUQf
Unstructured interviews |
Satisfaction; System information arrangement; Usefulness |
Median CSUQ score 2 (IQRg 1-3); 93% found app easy to use; 59% would use the app at home |
| Tsapepas et al [47] | Retrospective study | 282 | — | Self-created questionnaire | Satisfaction | Satisfaction rated 4 or 5 in 92% |
| Scott et al [48] | Prospective study (no control) | 20 | 14 days | SUS questionnaire; Semi-structured interview; Measurement of usage |
Satisfaction; System information arrangement; Usefulness |
Median SUS 95/100; Usage: 30% did not use after discharge |
| Warren-Stomberg et al [49] | Prospective study (no control) | 101 | 1 week | Measurement of usage | None | 55/101 used the app; Of those that used the app, 53% used >13 times out of possible 15 |
| Debono et al [50] | Prospective study (no control) | 60 | 15 days | Telephone interview | Satisfaction; Usefulness |
1 (poor) to 4 (excellent) scale: Overall satisfaction 3.4 Usability 3.5 Usefulness at home 3.2 Facilitating return at home 3.1; 91.6% would use the device again |
| Gunter et al [51] | Prospective study (no control) | 9 | — | SUS questionnaire | Satisfaction; System information arrangement |
Average SUS score 83.3/100; 55.6% were able to complete the tasks independently |
| Ponce et al [52] | Prospective | 31 | 24 days | 15-point questionnaire | Satisfaction; Usefulness |
Reassurance 4.6-4.8/5; Useful 4.5-4.8/5; Satisfaction 4.2-4.6/5 |
| Jiang et al [53] | Secondary retrospective analysis of previous RCT data | 96 | 12 months | Technology acceptance subscales used to measure: intention to use (1 item); perceived usefulness (4 items); and perceived ease of use (4 items) |
Satisfaction; Usefulness |
85% strongly agree with intention to use item; 80% gave high rating of perceived usefulness (>24/28); 82% gave high rating of perceived ease of use (>24/28) |
| Chai et al [54] | Prospective comparison study (nonrandomized) | 54 | 14 days | Self-created questionnaire | Satisfaction; Usefulness |
Satisfaction was >7.2/10 across all 4 items on questionnaire |
| Shellmer et al [55] | Prospective study | 7 | 6 weeks | 8/16 questions from PSSUQh survey | Satisfaction; System information arrangement; Usefulness |
Satisfaction 1/7 (1=strongly agree); Ease of use 1/7; Felt comfortable using application 1/7; “I could clearly tell when I missed my medication” 1/7; Liked tracking medications 3/7; Helpful to track medications 2/7 |
| Sun et al [56] | Prospective study | 66 | 30 minutes postoperation | Single question asked regarding preference of monitoring (app vs paper version of questionnaire) | Satisfaction | 76%-81% preferred the app over the paper version |
| Jaensson et al [57] | Prospective study | 10 | — | Self-created questionnaire on system layout and technical issues, satisfaction, and usefulness | Satisfaction; System information arrangement; Usefulness |
— |
| Symer et al [58] | Prospective study | 31 | 30 days | Measurement of usage; Self-created questionnaire |
Satisfaction; System information arrangement; Usefulness |
83.9% used the app 70% of the time; 89.3%: easy to navigate; 88.9%: easy to use; 85.2%: survey questions relevant for identifying problems related to readmission; 66.7% found reminders useful; 92.9% would recommend to others |
| Semple et al [59] | Prospective study | 65 | 30 days | Self-created survey; Interview; Usage |
Satisfaction | Satisfaction 3.7-3.9/4; 100% wiling to use in future; 100% surgeons found platform intuitive and easy to use; Usage: mean number of logins 19.3-23.9/30 days; Mean number of photographs uploaded 38-63/30 days |
| Bini and Mahajan [60] | RCT | 29 | 24 weeks | Self-created survey; Free-form feedback; Usage |
Satisfaction | Ease of use: 3.9-4.4/5; Satisfaction 4.2/5 |
aRCT: randomized controlled trial.
bSUS: System Usability Scale.
cuMARS: user version of the Mobile App Rating Scale.
dNot available.
eS-CAHPS: Surgical Care Consumer Assessment of Healthcare Providers and Systems.
fCSUQ: Computer System Usability Questionnaire.
gIQR: interquartile range.
hPSSUQ: Post-Study System Usability Questionnaire.