. 2023 Dec 9;31(3):746–761. doi: 10.1093/jamia/ocad222

Table 2.

The summary of the outcome measures nested in the seven main categories and subcategories in the review.

Category and subcategory (number of outcome measures, N_om)	Selected typical outcome measures (unit, questionnaire or method)
1. Functionality	(N_om = 44)
Sentence classification performance (N_om = 5)	Precision (%),²⁵^,⁹³ sensitivity (%),²⁵^,⁹³ accuracy (%),²⁵ and specificity (%),²⁵ and F1 (value)²⁵^,⁹³ of the classifier.
Understanding and responses (N_om = 17)	Response accuracy (%),¹⁷^,¹⁹^,⁸⁶^,¹⁰³, inquiries unable to answer (%),⁸⁶ response completion (%),⁸⁶ understanding (scale, survey),⁹¹ etc.
Engagement functions (N_om = 7)	Topics initiated by CA versus participants,²⁶ attempts to restart conversation (n),⁸ sentiment (score, coding responses manually),¹⁷ etc.
Task achievements and efforts (N_om = 4)	Conversation tasks completed (%),⁹⁴ task failure rate (n and %),⁸ and task completion (coefficient),²⁵ and time per task (seconds).⁸
Voice and device control (N_om = 2)	Adequate volume, speed, and sound quality (a survey),⁹¹ and negative technical aspects (qualitative analysis of user's responses).¹⁰⁴
Clinical assessment performance (N_om = 9)	Accuracy,⁴⁰^,¹⁰⁰ sensitivity,⁴⁰^,⁹⁰ specificity of CA-based clinical assessment outcomes (CA vs standard clinical assessments),⁴⁰^,⁹⁰ etc.
2. Safety and information quality	(N_om = 17)
CA response appropriateness (N_om = 6)	Response appropriateness (scale),¹⁹^,⁸⁶^,⁸⁷ appropriate responses (descriptive),⁸³ etc.
Risk of misinformation (N_om = 4)	Misinformation (%),¹⁷ reliable (%)⁸⁶ and evidence-base (%)⁸⁵ resources, information accuracy and completeness (%),⁸⁵ and quality (descriptive).⁸⁴
Risk of unintended harms or adverse events (N_om = 4)	Responses with risk of unintended harms (n and %; eg, medication and emergency tasks),⁸ serious adverse events (n),⁸² and deaths (n and %).⁸
Privacy and trust (N_om = 3)	Privacy and trust (a survey),⁵⁵ privacy and trust (a qualitative study, interview),⁵⁵ and privacy infringement (a survey).¹⁰⁵
3. User experience	(N_om = 80)
Ease of use (N_om = 2)	Ease of use (scale, a self-designed questionnaire)¹¹^,⁸⁸ and learning experience (score, a self-designed questionnaire).¹⁰⁶
Engagement (N_om = 3)	User engagement (scale, a survey),⁹⁵ DBCI engagement (scale),⁹² and perceived engagement (scale, a survey).¹⁰¹
Conversation capability (N_om = 6)	Response appropriateness (scale, a survey),¹¹ dialogue performance (score, SASSI),⁹⁴ emotional awareness (score, a questionnaire),¹⁰⁶ etc.
Usefulness/helpfulness (N_om = 6)	Usefulness (scale, a survey²¹^,¹⁰⁷ or interview⁸⁸), perceived helpfulness (a survey, open-ended question, or interview),²¹^,¹⁰⁸ etc.
Perceived quality and trust (N_om = 5)	Perceived trust (score, a questionnaire),⁸⁹ perceived quality of the answers (score, EORTC QLQ-INFO25),³⁹ etc.
Satisfaction (N_om = 5)	Satisfaction (scale, a self-defined survey,⁸^,¹⁵^,⁷²^,⁹⁹^,¹⁰⁵^,¹⁰⁷^,^109–111 and CSQ-8⁸¹^,⁸²^,¹¹²), content satisfaction (scale, a survey),¹⁰⁶ etc.
Feasibility (N_om = 3)	Feasibility (score, a self-designed questionnaire).¹⁸^,²⁰^,²⁶^,⁸¹
Usability (N_om = 5)	Usability (scale, SUS),⁷⁵^,⁹⁶^,¹⁰⁷^,¹¹³^,¹¹⁴ usability (open comments, a focus group session),²⁷ perceived usability (scale),⁹²^,¹⁰⁵ etc.
Acceptance/preference (N_om = 11)	Acceptance (scale, a survey),⁸⁸ preference of CA (scale, a survey),⁸⁸ potential to replace humans (scale, a survey),¹¹ etc.
Overall user experience with mixed themes (N_om = 26)	Overall user experience (UEQ,²³^,²⁵ USE,⁹⁶ NPS,¹¹⁵ URP-I⁸¹^,⁸²), users with positive or negative experience (n, the CA prompted the survey),¹⁵ etc.
Working alliance (N_om = 1)	Working alliance (questionnaire, WAI-SR⁷⁸^,⁷⁹^,⁸¹^,⁸²^,¹⁰¹^,¹¹²).
Suggestions for improvement (N_om = 4)	Suggestions for improvements (open-ended question in a survey),²⁰^,⁹³^,¹⁰⁸^,¹¹⁰ good and bad experiences with the CA (a survey),¹⁰⁹ etc.
Other open comments (N_om = 3)	Perceived stress (survey and interview),²⁴ benefit (focus group study),²⁷ and feelings of answering sensitive questions (CA vs humans).⁸⁹
4. Clinical/health outcomes	(N_om = 68)
Psychological/mental health (N_om = 34)	PHQ-9,²¹^,⁹⁵^,¹⁰⁶^,¹⁰⁹^,¹¹²^,¹¹⁶ QIDS-SR,¹⁰⁵, GAD-7,²¹^,⁸¹^,⁸²^,⁹⁵^,¹⁰⁶^,¹⁰⁹^,¹¹²^,^116–118 SAS,¹⁰⁵ PANAS,¹⁰⁶^,,¹⁰⁹^,¹¹²^,¹¹⁹ PSYCHLOPS,²¹ DASS21,⁷⁵^,¹¹⁷ PSS-10,⁹⁵^,¹⁰⁵^,¹²⁰ etc.
Disease conditions (N_om = 3)	Pain (%, NRS),⁸² Parkinson’s disease rating scale (MDS-UPDRS),¹²¹ and Parkinson’s disease questionnaire.¹²¹
Modification of behaviors and risk factors (N_om = 23)	Behavior modification (score, SQUASH),¹²²^,¹²³ smoking cessation (%, a survey),¹²⁴ physical activity (score, AAS),¹¹³^,¹²³ etc.
Knowledge and skills (N_om = 4)	Knowledge gained (scale, a survey),¹⁶^,¹⁸ problem solvability (score, a survey),⁷⁵ problem resolution (score, a survey),⁷⁵ etc.
Health wellbeing and issues (N_om = 4)	SWLS,¹²⁰ WHO-5-J,⁹⁵^,¹²⁵ EQ-5D-5L,¹²⁶ and falls (falls per 1000 patient-days).¹³
5. Costs and health economic analyses	(N_om = 2)
Cost effectiveness (N_om = 1)	Time spent per 100 patients (hours per 100 patients, an analysis of the conversation logs).¹⁴
Costs (N_om = 1)	Monthly budget (dollars per month, an analysis of running costs of the CA system).⁷³
6. Usage, adherence and uptake	(N_om = 62)
Usage (N_om = 38)	Conversation duration (second, minute, or hour),¹⁴^,¹⁵^,²⁶^,⁵⁵^,⁷²^,⁷³^,⁷⁵^,⁸¹^,⁸²^,⁸⁸^,¹¹⁵^,^126–128, exchanges (n),^108–110, CA responses (n),⁹²^,¹²⁷^,¹²⁹ etc.
Adherence (N_om = 15)	Adherence (n),¹⁰⁸^,¹²⁰ dropouts (n, %; conversation dropouts,¹⁵ and dropouts of interventions¹⁵^,¹¹⁶^,¹²²), follow-up rate (%),¹⁴ etc.
Uptake (N_om = 9)	Completed questionnaires (n),¹²² total followers (n),⁷³ total impressions (n),⁷³ average daily reach times (times of reach per day),⁷³ etc.
7. User characteristics for implementation science	(N_om = 12)
Age and gender (N_om = 2)	Age (age groups, n, %)¹⁵^,⁷²^,⁸⁸^,¹¹⁰ and gender (n, %).¹⁵^,⁷²^,⁸⁸^,¹¹⁰
Nationality, ethnicity and religion (N_om = 4)	Nationality (n),⁸⁸ race and ethnicity (%, White, Hispanic, Black),⁷² religion (n),⁸⁸ and language (%, users in Spanish).⁷²
Education and socioeconomic status (N_om = 2)	Occupation and education (n, %, self-designed questionnaire),⁸⁸ and urbanization levels (n, self-designed questionnaire).⁸⁸
Health conditions (N_om = 3)	Users with a personal history of cancer (%),⁷² a family history of cancer (%),⁷² and risks of different cancers (NCCN criteria, Tyrer–Cuzick criteria).⁷²
Devices used (N_om = 1)	Mobile users (%).⁷³

AAS, The Active Australia Survey; CSQ-8, Client Satisfaction Questionnaire with 8 questions; DASS21, depression, anxiety, and stress scales 21; DBCI, a questionnaire on the Digital Behavior Change Intervention; EORTC QLQ-INFO25, The European Organisation for Research and Treatment of Cancer Quality of Life Group information questionnaire; EQ-5D-5L, health-related quality of life with 5 dimensions: mobility, self-care, usual activities, pain/discomfort, and anxiety/depression; F1, the harmonic mean (average) of the precision and recall; GAD-7, the Generalized Anxiety Disorder scale—7; NCCN criteria, National Comprehensive Cancer Network criteria; NPS, net promoter score; PANAS, the positive and negative affect schedule; PHQ-9, the Patient Health Questionnaire—a 9-item self-report questionnaire that assesses the frequency and severity of depressive symptomatology within the previous 2 weeks; PSS-10, the Perceived Stress Scale; PSYCHLOPS, the psychological outcome profiles; ROC, receiver operating characteristic—a graphical plot to evaluate a binary classifier/decision system across different discrimination thresholds; QIDS-SR, the Quick Inventory of Depressive Symptomatology-Self-report; SAS, the Self-rating Anxiety Scale; SASSI, the Subjective Assessment of System Speech Interfaces—a 7-point Likert scale on accuracy, likeability, cognitive demand, annoyance, habitability, and speed; SQUASH, the Dutch Short Questionnaire to assess health enhancing physical activity; SUS, the system usability scale; SWLS, the Satisfaction with Life Scale; UEQ, the User Experience Questionnaire; URP-I, usage rating profile-intervention with feasibility (6 items) and acceptability (6 items) scales; USE, the Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire Short-Form; WAI-SR, the Working Alliance Inventory-Short Revised (agreement on the tasks of therapy, agreement on the goals of therapy and development of an affective bond); WHO-5-J, HEALTH well-being—5 Well-Being Index (Japanese version).