Table 2.
Task | Duration in seconds | Button clicks | Utterances | User errors (error rate)a | Chatbot errors (error rate)b |
---|---|---|---|---|---|
1. Find the list of current medications.c | 46.1 (25.4) | 1 (0) | 0 (0) | 0 | 0 |
2. Find 2 of the side effects of Amlodipine.c | 69.1 (25.3) | 3 (1) | 1 (1) | 0 | 0 |
3. Add 10 mg of Lisinopril every day at noon to your medication list.d | 131.6 (84.1) | 2 (1) | 3 (1) | 1 (10%)e | 4 (12.9%)f |
4. Update your medication reminder schedule for Amlodipine to remind you to take it on weekends at 10 am.d | 120.7 (52.6) | 4 (1) | 2 (1) | 1 (10%)g | 1 (4.5%)f |
5. View how many refills are left and the date of your next refill for Amlodipine.c | 60.8 (36.1) | 1 (1) | 1 (1) | 0 | 0 |
6. Find your blood pressure values for the month of August 2020.c | 54.1 (20.8) | 3 (3) | 1 (1) | 0 | 0 |
7. Add a blood pressure measurement of 120/80 that was taken on September 1, 2020.d | 100.1 (66.8) | 2 (1) | 3 (1) | 0 | 6 (23.1%)f,h |
8. Share your blood pressure measurements for the month of August 2020 with Dr. Smith.d | 69.3 (29.6) | 4 (1) | 1 (1) | 0 | 0 |
9. Schedule an appointment with Dr. Smith for Tuesday, November 17th at 1 pm.d | 43.8 (24.1) | 4 (2) | 1 (1) | 0 | 0 |
10. Find a healthy dinner recipe.c | 60.2 (28.8) | 2 (1) | 1 (1) | 0 | 0 |
Total (mean, SD) | 75.6 (51.5) | 2.5 (1.7) | 1.3 (1.2) | 2 (2.0%)a | 11 (8.6%)b |
User error rate was calculated per task by: number of participants who did not successfully complete the task/total number of participants.
Chatbot error rate was calculated per task by: number of unrecognized utterances/total number of utterances.
Data retrieval task.
Data entry task.
Type of user error: did not confirm medication was added.
Type of unrecognized error: date/time format.
Type of user error: inputted incorrect medication.
Type of unrecognized error: blood pressure format.