Approaches to Assess E-Health Programs: A Scoping Review

Abstract

Context:

With the increasing trend of adopting e-Health technologies, the need for evidence for assessing e-Health technologies has become crucial. The appraisal of the e-Health program is important as this could provide guidance on further e-Health investment and adoption.

Aims:

The aim of the study was to provide an articulated body of literature on the current state of knowledge about the assessment of e-Health interventions.

Settings and Design:

Scoping review was conducted based on the framework provided by Arksey and O'Malley (2005) and considering enhancements proposed by Peters et al. (2015).

Subjects and Methods:

We searched the electronic databases and available gray literature from inception until the last week of October 2019. PRISMA flowchart for the study selection process was used to guide reporting. Data extraction included information on study design, authors, year of study, country, and key findings in terms of approaches used for assessment of e-Health programs. Data were compiled and summarized narratively.

Results:

Searches were performed between October and December 2019. Seventy-one relevant papers published between the years until the last week of October 2019 were reviewed and analyzed. After considering all the eligibility criteria, 15 papers were included. We identified 15 approaches for assessing e-Health programs, which were summarized and tabulated.

Conclusions:

This review showed that available literature on the assessment of e-Health programs is heterogeneous in terms of the methodology used. e-Health interventions are highly contextual; therefore; the phase of maturity and objective of the assessment should be considered while carrying out the assessment of e-Health programs.

INTRODUCTION

The World Health Organization (WHO) defines e-Health as the use of information and communication technologies for health. In a broader sense, e-Health is about improving the flow of information through electronic means to support the delivery of health-care services and the management of the health systems. It encompasses a diverse set of informatics tools and processes that are designed with the view of improving public health and health care.[1]

Many countries including India are making significant investments in e-Health systems as e-Health has the potential to revolutionize the health-care delivery.[2] The digitalization of health system could play a catalytic role in improving the continuum of care. With the increasing trend of adopting e-Health technologies, the need for evidence for assessing e-Health technologies has become crucial.[3] Currently, e-Health programs are implemented based on assumptions of the benefits of e-Health only and there is a paucity of studies and documents which provide comprehensive guidance on assessing the e-Health programs.[4] The appraisal of e-Health program is important as this could provide guidance on further e-Health investment and adoption.[3]

SUBJECTS AND METHODS

This scoping review was conducted based on the framework provided by Arksey and O'Malley (2005) and considering enhancements proposed by Peters et al.[5] The protocol of the study was exclusively developed and prepared by using inputs from public health research experts. We used a PRISMA flowchart for the study selection process to guide reporting.[6] This review was conducted using the following steps: Identifying the research question, identifying relevant studies, study selection, charting the data, and collating, summarizing, and reporting the results.

Data sources and search

We searched the electronic databases and available gray literature from inception until the last week of October 2019. The search query consists of terms considered by the authors to describe the assessment of e-Health programs: e-Health, digital health, m-Health, assessment, and evaluation. The search strategy was not limited by the study design. An authentic database included English language literature indexed in Google Scholar, PubMed, and Scopus. We also searched for the articles included in the referencing list of the selected studies to avoid missing any literature. Other available literatures including academic papers dissertations, conference papers, and reports were also considered. This was done in a diverse format and audiences from other sources can provide invaluable information for our scoping review. Gray literature was also considered in this review as this type of literature could help significantly in reducing publication bias, increase reviews' comprehensiveness, and timeliness and foster a balanced picture of available evidence.

Study selection: Inclusion criteria

We developed explicit inclusion criteria for this review. The focus of this scoping review was to gather evidence on the practices of e-Health assessment. The Merriam Webster dictionary defines assessment as “the action or an instance of making a judgment about something: The act of assessing something.”[7] Hence, e-Health assessment is defined as an act of making a judgment about whether the e-Health program is functioning and producing the effects as expected. In this context, e-Health program can be any Information, Communication, and Technology (ICT)-based application, service, or resource used by an organization, providers, and consumers in managing health.[3] All studies conducted in any settings or country published in the English language were eligible for inclusion.

Study selection: Screening

Two reviewers independently reviewed titles and abstracts. Full texts of potentially relevant literature were sort and screened for inclusion criteria. Agreement between reviewers was achieved by common consensus. Conflicts were resolved by joint discussions with the third arbitrator reviewer. As this study was scoping review, no critical appraisal of included literature was performed.

Data abstraction

A review of full-text article and extraction of relevant data was independently done by two reviewers. Data extraction included information on study design, authors, year of study, country, and key findings in terms of approaches used for assessment of e-Health programs.

Data summary and synthesis

Data were compiled in a single spreadsheet. Considering the heterogeneity of the literature, we summarized the data narratively and did not pool the results of the included studies.

RESULTS

Searches were performed between October and December 2019. Seventy-one relevant papers published between the years until the last week of October 2019 were reviewed and analyzed. Nineteen duplicate papers were removed, 18 papers were excluded as they were not related to evidence on the practices of e-Health assessment, and 19 papers were removed as they were not related to ICT-based application, services used by providers, and consumers in managing health. After considering all the eligibility criteria, 15 papers were included. Figures 1 and 2 illustrate framework used for conducting scoping review and literature search process, respectively.

Figure 1.

Flowchart (framework) showing strategies for conducting scoping review. Adapted from-Scoping studies: towards a methodological framework (Hilary Arksey & Lisa O'Malley-2005)

Figure 2.

PRISMA flow diagram for literature search process

The scoping review revealed different viewpoints on the assessment of e-Health interventions. Different authors tried to provide a novel framework for the evaluation of e-Health programs. Tobias Kowatsch et al., proposed a framework for the design and evaluation of digital health interventions (DEDHIs).[8] This framework described life cycle of a DHI and recommended relevant evaluation criteria and implementation barriers to be considered for each phase of this life cycle. For each of the four phases of the DEDHI framework, corresponding evaluation tasks were mentioned. For the preparation phase, evaluation criteria were ease of use, adherence, personalization, safety, privacy, and security; for the optimization phase, evaluation criteria were effectiveness (individual components of the DHI), perceived benefit, content quality, personalization, perceived enjoyment, aesthetics, adherence, service quality, safety, privacy, and security; for the evaluation phase, evaluation criteria were effectiveness, perceived benefit, adherence, personalization, service quality, safety, privacy and security, and accountability; for the implementation phase, evaluation criteria were adherence, personalization, perceived benefit, content quality, ethics, service quality, safety, privacy and security, and accountability.[8]

Report of the Expert Panel on Effective Ways of Investing in Health, 2019, mentioned that most of the developed evaluation frameworks for evaluating digital health services were guided by health technology assessment (HTA) methodology.[9] Since HTA was developed in the context of curative health technologies, improvement in HTA methodology is needed in the context of evaluating digital health technologies. Since HTA does not address study aspects such as access, equity, patient empowerment, or goal orientation which are likely to be relevant in the context of digital health services, methods to systematically consider these aspects are needed to be developed for inclusion in HTA frameworks.[9] For evaluating the digital health services, border perspective is needed and tailor-made evaluation should be designed to capture all relevant changes in the appropriate manner. This report also recommended that to select an assessment strategy for digital health services, full description of the technology, its use and aim, intent to use, cost, consequences, and relevant comparators should be addressed. The development phase and implementation are crucial elements in evaluation.[9] The evaluation should focus on relevant outcomes in the context of the objective of the health system. All relevant cost benefits and broad impacts should be considered.

Høstgaard et al. developed a constructive e-Health evaluation method which offers a full evaluation framework to support and facilitate end-user involvement during the e-Health life cycle.[10] This method offers guidance through all life-cycle phases and a modified summative evaluation. It provides corresponding objectives, activities, and methods for phases, namely preanalysis, research and planning, design, development, implementation and diffusion, and modified summative evaluation. Data for this evaluation were collected using questionnaires, observations, interviews, and insight gathered from relevant documents.[10]

Biggs et al. identified priority areas for the development of the digital health benefit evaluation (BE) framework.[11] They provided an overview approach for BE of the digital health system in the context of My Health Record System. Five benefit workstreams such as customer and market insights; behavioral economics; data analytics; impact evaluations; and health economics evaluations were introduced to evaluate digital health initiatives using qualitative, quantitative, and mixed-method design.[11]

Murray et al. examined the challenges related to the assessment of the DHIs and outlined an evaluation strategy in terms of the research questions needed to appraise DHIs.[12] These research questions included defining the problem and the likely benefit of the DHI, the causal model describing how the intervention will achieve its intended benefit, key components and how they interact with one another, estimating overall benefit in terms of effectiveness, cost-effectiveness, and harms.[12]

Catwell and Sheikh proposed that e-Health interventions should be assessed while they are being designed, developed, and deployed.[13] They also emphasized that continuous systematic evaluations of e-Health interventions are needed.

Nabukenya and Justus Ashaba employed a qualitative approach to establish perspectives of key informants with respect to e-Health evaluation practices and challenges faced.[14] Most of the participants reported the system availability, system response speed, interoperability, usability, scalability, and availability of human resources to implement the e-Health initiatives as indicators that were currently considered during evaluations.[14]

Vukovic et al. showed that available HTA reports on e-Health/m-Health technologies are heterogeneous in terms of transparency and thoroughness.[15] These reports failed to tackle the relevant assessment elements, especially ethical, social, and organizational implications. They emphasized that HTA should provide a solid description of the technical, economic, clinical, legal, ethical, social, and organizational aspects related to the use of health technology.[15]

The WHO's (2016) guideline on monitoring and evaluating DHIs grouped evaluation of DHIs into four categories: (i) qualitative approaches, (ii) quantitative approaches, (iii) mixed-methods approaches, and (iv) financial and economic evaluations (EEs).[16] DHI projects typically begin with exploring basic questions of whether the intervention addresses the identified needs, including technical functionality and feasibility, followed by the assessment of user satisfaction, then move toward efforts to evaluate the effectiveness, attributable impact, and ultimately, “value for money” of the intervention.[16]

Lau and Kuziemsky (2017) in Hand Book of e-Health Evaluation described six empirical frameworks that have been used for the evaluation of e-Health systems.[3] These frameworks serve a similar purpose by providing an organizing scheme for e-Health practitioners to describe and predict the factors and processes that influence, design, implementation, use, and effect of e-Health system in a given health setting. BE framework, clinical adoption (CA) framework, CA meta-model framework, EE framework, pragmatic HIT evaluation framework, and holistic e-Health value framework were the proposed frameworks described in this handbook.[3] Table 1 illustrates the different approaches for e-Health assessment.

Table 1.

Suggested methods for e-Health evaluation/assessment

Suggested assessments methods/criteria	Proposed by	Description
DEDHI framework	Kowatsch et al.[8]	Evaluation criteria are based on the life cycle of DHI. For each of the four phases of DEDHI framework corresponding assessment task are provided[8]
Impact assessment of digital transformation of health services	EXPH (2019)[9]	Assessment of new digital services are based on health system goals, including quality, accessibility, efficiency, and equity as these goals are unaltered by the process of digitalization[9]
Minimum HTA inspired framework	Joint Action to Support the e-Health Network[17]	This framework provides four steps for assessing e-Health technology, namely preceding considerations, assess domains and issues, collect and analyze data, and report[17]
CeHEM	Hostgaard et al., (2017)[10]	This method provides a full life-cycle evaluation framework to support and facilitate end-user involvement. It involves methodology to guide the process during all phases with summative evaluation[10]
Digital health benefit evaluation frameworks	Biggs et al.[11]	This framework involves five various benefit workstreams to assess digital health initiative using qualitative, quantitative, and mixed-method design[11]
Research question-driven approach to the evaluation of DHI	Murray et al.[12]	This approach provided an assessment strategy in context to the research questions required to appraise DHIs[12]
Indicators for e-Health evaluations	Nabukenya and Ashaba[14]	This study used a qualitative approach to identify Indicators for e-Health evaluations. System availability, system response speed, interoperability, usability, scalability, and availability of human resources were identified as major indicators[14]
Continuous systemic evaluation	Catwell and Sheikh[13]	This framework provided a multifaceted, multidisciplined approach to facilitate continuous systematic evaluations throughout the life cycle of an e-Health intervention. This assessment aims to maximize the benefits and minimizing risks associated with the e-Health intervention[13]
Guideline on monitoring and evaluating DHIs	World Health Organization[16]	This guideline broadly grouped evaluation of DHIs into four categories: qualitative approaches; quantitative approaches; mixed-methods approaches; and financial and economic evaluations[16]
Infoway BE framework	Lau et al.[18]	This evaluation framework is based on Infoway BE framework which has three broad conceptual dimensions of HIS quality, use, and net benefits.[18]
CA framework	Lau et al.[19]	This framework is an extension of the BE framework which also involves contextual factors. The CA framework has three conceptual dimensions at the micro-, meso-, and macrolevels with each dimension having a set of factors that define e-Health success[19]
CAMM	Price and Lau[20]	The CAMM describes the CA of health information systems with four dimensions: availability, use, behavior changes, and outcome changes. These dimensions are dependent on each other and should be considered collectively for planning an evaluation[20]
e-Health value framework	Lau and Kuziemsky[3]	This framework demonstrated how the value of e-Health is influenced by investments, dynamic interactions among contextual factors for adoption, and the lag times for adoption and impact[3]
e-Health economic evaluation framework	Lau and Kuziemsky[3]	This framework provides a classification scheme to understand various approaches used in e-Health economic evaluation studies and has six components: Perspective, options, time frame, input costs, outcomes, and analyzing/comparing options[3]
Pragmatic HIT evaluation	Lau and Kuziemsky[3]	This framework provides guidelines for promoting consistency and quality in the process evaluation, reporting dissemination of e-Health intervention[3]

DEDHI: Design and evaluation of digital health interventions, DHI: Digital health intervention, HTA: Health technology assessment, CeHEM: Constructive e-Health evaluation method, CA: Clinical adoption, BE: Benefit evaluation, CAMM: Clinical adoption meta-model, HIT: Health information technology

DISCUSSION

In this scoping review, we performed a rigorous search of the literature to provide a narrative description of various approaches suggested to assess e-Health programs. We identified 15 types of approaches that were suggested by various authors. This review showed that available literature on the assessment of e-Health programs is heterogeneous in terms of the methodology used. We tried to provide a summary and tabulated these diverse approaches to provide an articulated body of literature on the current state of knowledge about the assessment of e-Health interventions. Assessment of e-Health can be carried out using a single approach or a combination of different approaches. The approach can be formative/summative, objective/subjective, and qualitative/quantitative.[3,16] Formative assessment is conducted during the time of implementation. Information acquired through it could be used to shape the activity as it happens, but in contrast, summative assessment is conducted at the end of the implementation phase and is used to make decisions about the success or failure. Constructive e-Health evaluation presented by Høstgaard et al., provides an illustration of a full evaluation framework for the entire e-Health development life cycle involving both formative and summative assessments.[10] According to the WHO, formative evaluations should be carried out internally by the organization itself so that its findings can be utilized to influence program plan and implementation. Summative evaluations, on the other hand, should be conducted by an external body so that it could help to generate evidence important for improvement, implementation, and expansion of the program.[16] Objective assessment is described by measurable properties that are generally independent of the observer, whereas subjective assessment is based on the perception of the observers or users.[16] DEDHI framework which provides assessment criteria based on the life cycle of DHI includes both subjective (e.g., perceived benefit of a DHI) and objective (e.g., adherence to a DHI) evaluation criteria. Qualitative assessment can be used to assess user satisfaction and border context of system workflow, whereas quantitative assessment tends to measure changes in process, outputs, and outcomes.[8] These approaches were used by digital health benefits evaluation frameworks which assessed the five benefits workstream of the My Health Record System in Australia using qualitative, quantitative, and mixed-method design.[11]

Assessment can also be performed considering organizational, technological, social, clinical, ethical, and economic aspects. These aspects can be considered during the various life-cycle phases of e-Health intervention and could differ in each phase depending on the activities of the phases.[4] Continuous Systemic Evaluation proposed by Catwell and Sheikh from Centre for Population Health Sciences, Scotland, tried to provide a multifaceted and multidiscipline assessment approach to evaluate e-Health interventions taking account of political, social, organizational, and technical contexts.[13] Various frameworks and approaches discussed in this review have its own strengths and limitations. For instance, the minimum HTA inspired framework covers most of the context mentioned above, but it is unable to consider interoperability, data quality, and accessibility and usability criteria.[17] Similarly, the Infoway BE framework which assesses e-Health on three broad conceptual dimensions of health information system quality, use, and net benefits does not cover organizational and contextual factors.[18,21]

It is very difficult to evaluate all possible outcomes and to develop a standardized single approach to assess e-Health programs and intervention. This might be attributed to the complex nature of the health system and evolving health technologies. This review has a few limitations. Non-English language studies were not considered in this review as researchers were lacking competency in using the non-English language studies, quality appraisal of studies was not carried out and even after doing a comprehensive search, only 15 studies were identified which fulfilled the inclusion criteria.

CONCLUSIONS

This review showed that available literature on the assessment of e-Health programs is heterogeneous in terms of the methodology used. e-Health interventions are highly contextual; therefore; the phase of maturity and objective of the assessment should be considered while carrying out the assessment of e-Health programs.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.