. 2023 Dec 9;31(3):746–761. doi: 10.1093/jamia/ocad222

Table 3.

The proposed framework for evaluating CAs in healthcare.

	Stage →	1. Feasibility and usability →	2. Efficacy →	3. Effectiveness →	4. Implementation
WHO digital health	Brief description	Feasibility: The ability to work as intended. Usability: The degree of a system being used to achieve specified goals in a specified context of use.	Efficacy: The ability to achieve the intended results in a research setting or trial.	Effectiveness: The ability to achieve the intended results in a real application (nonresearch setting).	Implementation science: To assess the uptake, integration and sustainability of evidence-based digital health interventions for a given context, including policies and practices.
	Evaluation targets	Stability (system uptime/failure rates) Performance consistency Standards adherence (terminology, interoperability, security)	User satisfaction Workflow “fit” Learning curve (design) Cognitive performance/errors Reliability	Changes in care processes (time) Changes in outcomes (system performance/health)	Changes in process, outcome, coverage, and costs Total cost of implementation, and health impact Error rates Learning curve of users Changes in policy, practices attributable to system Adaptability and extendibility to new use-cases
	Illustrative num of users	10-100	100-1000	10 000 +	1000 000+
Studies reviewed and outcome measures at 4 major evaluation stages aligned with the WHO guide	Studies (n)	40	21	12	8
	Study design and sample size (n)	Single-arm studies (n = 3-10,¹⁰⁷^,¹¹⁴^,¹³¹ 11-20,²¹^,²⁷^,⁹¹^,⁹⁵^,⁹⁷^,¹⁰³^,¹¹¹ 21-30,¹¹^,²⁰^,²³^,²⁴^,²⁶^,⁹⁶ 31-40,²⁵^,⁸⁰^,⁹⁰ 41-50,⁸^,⁵⁵^,⁹⁸^,¹⁰⁵, 73,⁹² 89,⁹⁴ 101,⁸¹ 116,¹¹³ 121,⁹³ 318⁷⁴; 4390 messages⁷³), 2-arm quasi-experimental study(n = 454¹²⁵), laboratory tests (investigators, n = 1,¹⁹ 2¹⁷^,^83–87) and RCT (n = 142,³⁹ and 289¹³²)	Single-arm studies (n = 28,¹⁰⁸ 31,¹²³ 44,¹⁸ 47,⁸⁹ 61,⁷⁷ 128,⁹⁹ 139,¹⁰⁰ and 154²²) Case-control study (n = 153,⁷⁶ and 270¹⁴) Randomized controlled trials (n = 20,¹²¹ 23,¹¹⁹ 70,¹⁰⁶ 74,¹⁰⁹ 83,¹¹² 112,⁷⁵ 153,¹⁰¹ 197,¹¹⁵ 181,¹¹⁶ 210,¹⁰² and 958¹²²)	Cross-over study (n = 179⁴⁰) Case-control study (n = 95¹³) Cross-sectional studies (n = 354,¹²⁷ 929,⁸⁸ 4737¹¹⁰) Randomized controlled trials (n = 28,¹²⁰ 180,⁸² 513,¹²⁶ 700,¹¹⁸ 927,¹⁶ 57 214¹²⁴) Cohort study (n = 3629¹¹⁷)	Cross-sectional studies (n = 1206,⁷⁸ 7099,¹⁰⁴ 16 519,¹⁵ 14 698,¹²⁹ 36 070,⁷⁹ 61 070,⁷² 135 263¹²⁸; 610 conversations¹³³)
	User characteristics for implementation science	Devices used.⁷³	–	Age and gender⁸⁸^,¹¹⁰ Nationality, ethnicity and religions⁸⁸ Education and socioeconomic status⁸⁸	Age and gender¹⁵^,⁷² Nationality, ethnicity and religions⁷² Health conditions⁷²
	Usage, adherence and uptake	Usage, ²¹^,²⁶^,⁵⁵^,⁷³^,⁸¹^,⁹²^,⁹⁸^,¹⁰³^,¹¹³^,¹¹⁴^,¹³¹ Uptake²⁷^,⁷³	Usage,¹⁴^,⁷⁵^,⁷⁷^,¹⁰⁸^,¹⁰⁹^,¹¹⁵^,¹¹⁶^,¹²¹^,¹²³ Adherence¹⁴^,¹⁰⁸^,¹¹²^,¹¹⁶^,¹¹⁹^,¹²⁰	Usage,⁸²^,⁸⁸^,¹¹⁰^,¹¹⁷^,¹²⁴^,¹²⁶^,¹²⁷ Uptake¹²² Adherence¹²²	Usage,¹⁵^,⁷²^,¹²⁸^,¹²⁹^,¹³³ Uptake⁷²^,¹²⁹ Adherence¹⁵^,⁷²^,¹³³
	Costs and health economic analyses	Costs⁷³	Cost effectiveness¹⁴	Health economic analyses, such as cost-utility analysis, cost-effective analysis, cost-minimization analysis, and cost-benefit analysis. (our recommendation)	Health economic analyses, such as cost-utility analysis, cost-effective analysis, cost-minimization analysis, and cost-benefit analysis. (our recommendation)
	Clinical/health outcomes	Knowledge and skills²¹ Health wellbeing and issues⁹⁵^,¹²⁵ Psychological/mental health,²¹^,⁵⁵^,⁷⁴^,⁸¹^,⁹⁵^,¹⁰⁵^,¹²⁵ Behavioral modification and risk factors²⁶^,⁸¹^,¹¹³	Disease conditions¹²¹ Knowledge and skills¹⁸^,⁷⁵ Health wellbeing and issues¹²⁰ Psychological/mental health.^75–77^,¹⁰²^,¹⁰⁶^,¹⁰⁹^,¹¹²^,¹¹⁵^,¹¹⁶^,^119–121 Clinical assessment performance,²²^,⁸⁹^,¹⁰⁰ Behavioral modification and risk factors¹⁰⁸^,¹²³^,¹⁰⁰	Disease condition⁸² Knowledge and skills¹⁶ Health wellbeing and issues¹³^,¹²⁶ Psychological/mental health¹³^,¹⁶^,⁸²^,¹¹⁷^,¹¹⁸ Clinical assessment performance⁴⁰ Behavioral modification and risk factors¹⁶^,⁸²^,¹¹⁰^,¹¹⁸^,¹²²^,¹²⁴^,¹²⁶	Psychological/mental health⁷⁸^,⁷⁹ (through short inbuilt questionnaires in CA apps)
	User experience	Usability²⁷^,⁹²^,⁹⁶^,¹⁰³^,¹⁰⁵^,¹⁰⁷^,¹¹³^,¹¹⁴ Feasibility²⁰^,²⁶^,⁸¹ Engagement⁹²^,⁹⁵ Ease of use¹¹ Satisfaction⁸^,⁸¹^,⁹¹^,¹⁰⁵^,¹⁰⁷^,¹¹¹ Open comments²⁴^,²⁷ Working alliance⁸¹ Overall experience⁸^,²¹^,^23–25^,⁵⁵^,⁷⁴^,⁸⁰^,⁸¹^,⁹²^,^94–98^,¹⁰⁵^,¹¹³^,¹¹⁴^,¹³¹^,¹³² Usefulness/helpfulness²¹^,⁵⁵^,¹⁰⁷ Acceptance/preference¹¹^,⁸¹^,⁹⁸^,¹⁰⁵^,¹³² Conversational capability¹¹^,²³^,⁹⁰^,⁹⁴^,¹⁰⁷ Perceived quality and trust³⁹^,⁹⁷^,¹³² Suggestions for improvement²⁰^,²⁴^,²⁷^,⁹³	Usability⁷⁵^,¹¹⁵ Feasibility¹⁸ Ease of use¹⁰⁶ Satisfaction⁹⁹^,¹⁰⁶^,¹⁰⁹^,¹¹² Engagement¹⁰¹ Working alliance¹⁰¹^,¹¹² Overall experience⁸⁹^,⁹⁹^,¹⁰⁶^,¹⁰⁸^,¹¹²^,¹¹⁵^,¹²⁰ Other open comments⁸⁹ Acceptance/preference¹⁸^,⁷⁵^,⁸⁹^,¹⁰⁰^,¹²² Usefulness/helpfulness¹⁰⁸^,¹¹⁶ Conversational capability¹⁰⁶ Perceived quality and trust⁸⁹ Suggestions for improvement¹⁰⁸^,¹⁰⁹	Ease of use⁸⁸ Satisfaction⁸²^,¹¹⁰^,¹²² Working alliance⁸² Overall experience¹⁶^,⁸²^,¹¹⁰^,¹²⁰ Usefulness/helpfulness⁸⁸^,¹¹⁸ Acceptance/preference⁸²^,⁸⁸ Conversational capability⁸⁸ Suggestions for improvement¹¹⁰	Satisfaction¹⁵^,⁷² Working alliance⁷⁸^,⁷⁹ Overall experience¹⁵^,⁷⁸ Acceptance and preference¹⁰⁴
	Safety and information quality	Privacy and trust⁵⁵^,¹⁰⁵ Risk of causing death⁸ CA response capability¹⁷^,¹⁹^,⁸⁵^,⁸⁶ Risk of misinformation¹⁷ Risk of unintended harms⁸^,⁸⁵ CA response appropriateness¹⁹^,⁸³^,⁸⁶^,⁸⁷^,¹⁰⁵ Resources and contents quality^84–86		Risk of unintended harms⁸²
	Functionality	Response speed⁸ Task achievements⁸^,²⁵^,⁹⁴ Engagement functions⁸^,¹⁷^,²⁶^,⁸⁵^,⁹⁴ Voice and device control⁹¹ Classification performance²⁵^,⁹³ Clinical assessment performance⁹⁰ Understanding and accurate responses¹¹^,¹⁹^,⁸⁶^,⁹¹^,⁹⁸^,¹⁰³^,¹⁰⁷^,¹¹¹^,¹³²	Understanding and accurate responses¹⁰⁸	–	Voice and device control¹⁰⁴ Understand and accurate response¹³³

The framework demonstrates the included CA evaluation studies (n = 43), study designs, outcome measures and sample sizes at four essential evaluation stages. The 4 stages and corresponding evaluation targets were proposed by the WHO. Essential categories, which we identified for each stage, are marked by a light blue.