Evaluation Methods in Clinical Health Technologies: A Systematic Review

Abstract

Background:

This study was conducted to classify the types of evaluation methods in clinical health technologies based on a systematic review method.

Methods:

An electronic search was conducted in three scientific databases including Scopus, PubMed and ISI. The search strategy was performed in Jul to Nov 2021 and based on the three main concepts of “evaluation”, “technology”, “health. This search has been restricted to 10 years (2011–2021). Moreover, it only was limited to English and papers published in journals and conferences proceeding.

Results:

Overall, 8149 references were identified for title and abstract screening. Full text screening was performed for 2674 articles, with 174 meeting the criteria for study inclusion.

Conclusion:

Most of the technologies evaluated in these articles were associated with PC-based systems (N=107), and there have been fewer mobile apps (N=67). Most of used technologies were with goals of treatment (43%, N=74) and education (26%, N=45). Among all the methods, the most and the least used methods were usability (66%, N=115) and qualitative (1%, N=2) method, respectively. The most method for health clinical technologies is usability method especially in telemedicine field.

Introduction

The health info systems field contains a distinctive chance to find out from and extend the work that has already been done by the extremely correlative info systems field (1). Providing high-quality healthcare services can promote public health significantly (2). Developments in information technology (IT) and information systems (IS) fields are changing the healthcare industry (3). Institute of Medicine (IOM), based on multiple studies citations, emphasized that information technology and information systems can play an important role in providing safe, timely, effective, and efficient healthcare services (4,5).

The health care environment has evitable changes due to information technologies developments (3). In a connected care setting, additional voters are participating in their health care through mobile apps and social media tools (6). There's a developing drift of social moving from computer proficiency to advanced wellbeing education that consolidates information and abilities related to the utilize of computerized devices in our associated wellbeing care environment (7).

Digital technology for screening, treatment, and management of health has proliferated in recent years, especially patients' empowerments, and their involvement in disease management are increased rapidly by digital technologies (8–10). Patient self-tracking or recording of health indicators reception has been utilized in a spread of things, as well as the prediction of events like migraines, weight management, physical activity patterns, and self-management of vital sign and blood sugar (11–13).

With the recent advances in data and technology (ICT), we have a tendency to area unit getting into a “superconnected society,” wherever technology has been employed in the health domain for varied functions, like for storing electronic health records, monitoring, education, communication, and activity trailing (11). E-Health, CDSS, patients portals, CPOE, Telemedicine, and other technologies in healthcare contexts, and on the other hand Mhealth is another promising technology that contributes a considerable value in delivering health care services represents an extraordinary opportunity to achieve all of these features (14).

The Electronic Health Record Sharing System (E-Health) provides health records in an electronic format that contain health-related information of people. eHealth provides an economical platform for aid suppliers to transfer and access somebody's health-related information (15). Clinical decision support systems (CDSS) square measure computer-based programs that analyze information among EHRs to produce prompts and reminders to help health care suppliers in implementing evidence-based clinical pointers at the purpose of care (16). A patient portal may be a secure on-line web site that offers patients convenient, 24-hour access to non-public health data from anyplace with an online association. (17).

Healthcare technologies demand analysis could be a well-functioning routine health data system that ensures the assembly, analysis, dissemination, and use of reliable and timely data associated with service delivery (18,19). Evaluation methodologies and measures adapted from many disciplines; focus on process (20). Additionally, Digital health interventions unit scalable tools to spice up health and provision by rising effectiveness, efficiency, accessibility, safety, and personalization (21).

In a word, through the analysis of data and communications instrumentality, will establish and improve the state of data and communications instrumentality analysis system, deepen the knowledge facility check, state analysis, technical oversight, technical analysis, then on every work, improve the standard of analysis suggests that and instrumentality quality closed-loop management mechanism, solid foundation data facility quality in associate comprehensive method (20,22).

Information systems are extensively used in various healthcare settings and have improved the quality, efficiency, and effectiveness of health services and overall patient satisfaction (23,24). Given this growing health care engagement, it is important for health care professionals to have the knowledge and skills to evaluate and recommend appropriate digital tools (6).

Evaluation suggests the making of a judgment regarding the quantity, number, or worth of one thing (25). In another study, even the word “evaluation” means different things to researchers from different disciplinary backgrounds (20). Effective evaluation allows us to understand how and under what conditions system or application work, and determine the safety and effectiveness of the system (26,27).

Evaluation has outcomes at both the policymaking and service delivery levels (28,29). Evaluation can guide the implementation process and mitigate unplanned negative outcomes (30,31). A wide variety of methods is available to evaluate health clinical technology systems (32). In a category argued about three main categories in health systems and technologies refer to process, outcome, and impact evaluation and most studies evaluated the process stage in system design and development such as usability methodology (33). One category classified evaluation methods to two main category as formative and summative (34).

In another classification researcher divided evaluation method to Financial and Nonfinancial (35) which in this study, evaluating the financial field was not the purpose of the study. There are unit incommensurable philosophy variations between analysis traditions (28). Performance Appraisal area unit typically usually classified into twin groups: ancient (Past oriented) ways and modern (future oriented) ways and different researchers have classified the existent ways to some groups; absolute standards, relative standards and objectives.(36).

According to Si Chen and et al. “Evaluation research can be defined as a form of “disciplined inquiry”, which “applies scientific procedures to the collection and analysis of information about the content, structure and outcomes of programmes, projects and planned interventions”” (37).

One of the most important usability factors in previous studies is the user of the technology and its field of use. In this systematic review, different (quantitative and qualitative) methods of evaluating health technology-based systems were studied. The goal of this research was to classify the types of evaluation methods in clinical health technologies and classify their utility in each area.

Materials and Methods

Search Strategy

This review was reported according to a systematic review protocol to study different methods of evaluating health technology-based systems. At the first step, an electronic search was performed in three scientific databases including Scopus, PubMed, and Web of Science. The search strategy was based on the three main concepts of “evaluation”, “technology”, “health”. The search terms used were not restricted to the title only. They were found within the MeSH term, title, abstract, or keywords. The steps for making a search query in PubMed are listed in Table 1.

Table 1:

Search query in PubMed

1	("Computer Systems"[Mesh] OR "Computer System*"[Title/Abstract] OR "Mobile Applications"[Mesh] OR "Mobile App*"[Title/Abstract] OR "Software"[Mesh] OR "Software"[Title/Abstract] OR "Information Systems"[Mesh] OR "Information System*"[Title/Abstract] OR "Informatics"[Mesh] OR "Informatic*"[Title/Abstract] OR "portal" [Title/Abstract] OR "web"[Title/Abstract] OR "electronic device"[Title/Abstract] OR "information technology" [Title/Abstract])
2	("Health"[Mesh] OR "Disease"[Mesh] OR "Medicine"[Mesh] OR "Therapeutics"[Mesh] OR "Diagnosis"[Mesh] OR "Delivery of Health Care"[Mesh] OR "health" [Title/Abstract] OR "disease" [Title/Abstract] OR "therap*" [Title/Abstract] OR "diagnosis"[Title/Abstract] OR "medicine" [Title/Abstract] OR "illness" [Title/Abstract] OR "sickness" [Title/Abstract] OR "well-being" [Title/Abstract] OR "fitness" [Title/Abstract] OR "wellness" [Title/Abstract] OR "normality" [Title/Abstract] OR "medical" [Title/Abstract] OR "treatment" [Title/Abstract])
3	("Eval*"[Title/Abstract] OR "assess*"[Title/Abstract] OR "appraisal"[Title/Abstract] OR "examination"[Title/Abstract] OR "validation" [Title/Abstract])
4	1 AND 2 AND 3

In other search databases, equivalent searches were performed by the rules of that database (Appendix A). The searches were performed in Jul 2021. Figure 1 illustrates the PRISMA process for data collection and analysis.

Fig. 1:

Study Flow Diagram

Inclusion Criteria

This search has been restricted to 10 years (2011–2021). Moreover, it only was limited to English and papers published in journals and conferences proceeding. Since our focus is on evaluation methods of health technologies, financial, cost effective and cost benefit evaluations were excluded.

Exclusion Criteria

Papers that were not available in full text (First, we send an email to corresponding author for access the full-text. In the final step, if we didn’t access to full-text, we exclude them), non-human monitoring, non-clinical purposes such as financial, review articles, and RCT Studies, excluded. Papers that study non-human monitoring and for non-clinical purposes and review papers were excluded.

Screening and paper selection

Searched papers were imported into Endnotes reference management software ver. 8.1. After removing duplicate records, the papers were independently evaluated on the title and abstract level by two reviewers (M.R and R.H). Experts extracted papers information in five categories, including type of product, usage goal, area of use, type of user, and type of evaluation method. Disagreements between reviewers were settled by consensus, or in consultation with third and fourth reviewers (N.M and S.P).

Then, the full texts of the papers were downloaded for further review. This review was reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Fig. 1). The majority of papers were deleted for repetition and lack of availability of full text, and 174 papers remained for review (Fig.1).

Results

The search retrieved 174 papers from the third scientific database. Most of the health technologies evaluated in these articles were related to PC-based systems (N=107), and there were fewer mobile apps (N=67). In the following, descriptive statistics of the results in the form of tables and figures are presented:

The most common area of technology use was the telemedicine (61%, N=106) and decision support systems (15%, N=26). And the lowest frequency was in three items, CPOE (1%, N=2), Virtual reality (1%, N=2), and Registry (2%, N=3), respectively. Also, most of the used technologies were with goals of treatment (43%, N=74) and education (26%, N=45) (Table 2).

Table 2:

Frequency of product type under evaluation & area of use according to the usage goal

Variables		Diagnosis		Education		Follow-up		monitoring		prevention		Rehabilitation		Screening		Treatment		Total
		N	%	N	%	N	%	N	%	N	%	N	%	N	%	N	%	N	%
Type of product	App	6	55	28	62	4	57	4	36	2	100.00	2	40.00	5	26	16	22	67	39
	PC-based system	5	45	17	38	3	43	7	64	0	0.00	3	43	14	77	58	78	107	61
	CPOE	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	2	3	2	1
	DSS	6	55	1	2	1	14	2	18	0	0.00	0	0.00	5	26	11	15	26	15
	E-H-R	0	0.00	1	2	0	0.00	1	9	0	0.00	0	0.00	1	5	9	12	12	7
	HIS	0	0.00	2	4	1	14	0	0.00	0	0.00	0	0.00	0	0.00	20	27	23	13
	Registry	1	9	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	1	5	1	1	3	2
Area of use	Telemedicine	4	36	41	91	5	71	8	73	2	100.00	5	100.00	12	63	29	39	106	61
	VR	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	2	3	2	1
Total		11	6	45	26	7	4	11	6	2	1	5	3	19	11	74	43	174	100

As our result, the highest frequency of health technologies is created for patients (N=75) and the lowest for healthy people in society people (N=3). Moreover, the only useful where the number of apps is more than PC-based systems, it is related to patient’s N for apps=41 and N for systems=34. About the types of evaluation methods used in the included papers, among all the methods, the most and the least used methods were usability (66%, N=115) and qualitative (1%, N=2) methods, respectively. The most used evaluation method is quantitative and the most method in this category is usability method.

According to Fig. 2, technologies used for physicians are mostly with the goal of treatment (N=9), and for patients with the goal of education (N=35) and treatment (N=21).

Fig. 2:

Frequency of technologies used according to usage goals & users

As can be seen in Fig. 3, most evaluations were of the usability type performed on telemedicine (N=71).

Fig. 3:

Frequency of technologies used according to usage goals & users

Discussion

The health system scope environs have a characteristic chance to build on and support the work that has so far been accomplished by the hardly interconnected IS field. Our findings mentioned higher than give a robust summary of what has been worn out the health system evaluation field and determine areas for future researches for the analysis of health care systems. Some of these systems could not obtain their predetermined aims or have not been well adopted (38). For instance, some systems may beginning new types of errors (39) or needs more time from the providers to accomplish their duties with the utility of these systems (40,41).

All evaluation methods can be classified into two main methods: quantitative evaluation and qualitative evaluation (34,35,37). Both quantitative and qualitative ways, and maybe a mixed-methods approach, may be adopted in analysis. One in every of the foremost prevailing and elementary classifications between styles of analysis was introduced by Scriven in 1967 as acknowledged by Clarke (37). The most common evaluation ways tools in quantitative ways are Think-Aloud (testing analysis method), heuristic technique, cognitive Walkthrough, Usability analysis, skilled Reviews, Focus teams (Inquiry analysis method), Scenarios, and review (Inquiry analysis technique) used supported method and scientist need (42). Usability evaluation has a significant role to play when conditions warrant it especially in the process stage of system development and implementation (32). Classification criteria of usability evaluation methods for interactive adaptive systems formerly presented in a research paper as usability factors, evaluation phase, adaptation layer, stakeholders, location, requirement resources, and advantages and disadvantages (43).

The former researcher divided the system evaluation method into 9 categories as Heuristic evaluation, cognitive walkthrough, task analysis, GOMS analysis, usability testing, field study, structured interview, think-aloud method and multiple and said that heuristic evaluation and cognitive walkthrough are the most popular expert-based UEM in the healthcare domain (44). In this think about, frameworks assessment strategies were isolated into 6 major categories, counting heuristic, subjective, ease of use, ease of use and client fulfillment, client fulfillment, and different assessments. And in another ponder, these strategies were categorized into 13 classes(45).

Usability is a critical element for interactive adaptive system achievement. The approach taken to usability in ISO 9241-11 (1998) was similar, defining usability as: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (46). A number of methods have been proposed to patronage the usability evaluation of interactive adaptive systems. Several functions for the usability evaluation of clinical Systems have been proposed already (28).

During the last two decades in the scope of human-computer interaction, usability dealt with how to suit a user interface (UI) has been developed, so as if it may be utilized by end-users simply, efficiently, and with a sense of well-being (47). Ease of use issues are counted among the most reasons for these insufficiencies and it is exceedingly prescribed to persistently assess the ease of use of the frameworks to be able to recognize and handle these issues (4,38,48). Poor usability can be beginning decreased efficiency and effectiveness prominent to reduced reliance on the system and users’ displeasure. Systems with usability defects may enhance mistake potential and even conducted to unrecoverable disasters (49). The usability method is 66% more common than other methods, while the heuristic method is 3 percent. Heuristic evaluation is a widely popular method for discovering the sources of the trouble of usability problems. In this method, expert evaluators examine a user interface using a set of criteria (33,43). One of the most common usability evaluation methods is heuristic evaluation (4). Moreover, heuristic evaluation was one of and the most commonly used usability evaluation method (50).

Technology tools based on our research can be classified into 6 main groups, includes CPOE, CDSS, EHR, HIS, registry, and telemedicine. EHRs are real-time, patient-centered records that build info accessible instantly and privacy to approved users, and at first, CPOE systems were marketed and sold-out as standalone systems, however currently a lot of electronic health record (EHR) merchandise contain CPOE modules that permit physicians to enter patient information electronically into text boxes and drop-down menus, instead of written notes (51,52). Hersh noted that there were 450 telemedicine programs available worldwide at the time of writing (53). In our research, most evaluation has been done on telemedicine systems (N= 106). And the most method for evaluation in the tele-medicine field was the usability evaluation method with 71 cases in return qualitative method with 1 item. Health systems use for diagnosis, training, follow-up, supervising, prevention, rehabilitation, screening, and treatment, and nowadays, to handle patients as a supplement has elevated (54,55). And our findings show this with 74 studies of 174 according to table number 2.

Mobile health gadgets are used for monitoring the connection among treatment agents with an interactive data imagination approach and are functional and trustworthy tools for sick people (55). In this research users of new technologies are categorized into 6 general groups includes clinicians, nurses, patients, physicians, people, and multiple. In other studies, technology users to classified 6 categories too but they considered it as physicians, nurses, biomedical engineers, technicians, administrators, and patients (56,57). As our findings showed, several of these tools are designed for patient utilization (43%). Of these, 41 were applications and the rest were health information systems.

Information systems can refine cost control, grow the timeliness and precision of patient care and managing information, increase service capacity, lessen personnel costs and inventory levels, and improve the standard of patient care. Therefore, evaluation is very important in attention as a result of it supports an evidence-based approach to apply delivery and, it's accustomed assist in decision making however well one thing is functioning. It will inform choices concerning the effectiveness of service and what changes might be thought-about to enhance service delivery and its application for end-users.

In this study, articles were introduced that were summative versus formative, evaluating the system after implementation. Formative evaluation method aims to provide systematic feedback to the implementers while summative evaluation is concerned with identifying and assessing the worth of program outcomes in the light of initially specified success criteria after the implementation of the change program is completed (34). Based on many benefits of the formative evaluation method suggested that both formative and summative evaluation methods be considered.

Implications for policy makers

The obtained results provide valuable information and knowledge for health policy makers in the application of clinical systems after evaluation to help technology developers. In addition, it will be key to improving digital health and differentiating health information systems.

Limitations

In this study, the significant limitation faced by the researchers was the lack of access to the databases in different time periods, which forced several steps to re-search so that no data is lost.

Conclusion

Healthcare organizations are complex and under some pressure to integrate technology into their practice to transform care and become more efficient. Technology is a powerful tool for diagnosing, treating, tracking and monitoring patients, so today, physicians, nurses, clinicians, and patients have shown great attention to the use of new treatment technologies. We examined the overall health clinical technology systems and applications were published in leading evaluation methods. Innovative solutions are needed to reduce information overload and ensure users are empowered to make informed decisions about their health care. Technology can be an important lever in people's health, but these tools can be used with confidence if they are properly evaluated. In this study, we did not find a study that demonstrates the content evaluation of the system. Rarely is a system or clinical technology evaluated by its outcome or impact. In this study, the least evaluation has been done on the technologies used in prevention and rehabilitation. The most evaluation of health technologies is usability evaluation. Most importantly, our analysis points to the need for further attention to system evaluation basically in the content field, outcome, and impact evaluation. Moreover, suggested employing multiple methods or mixed evaluation methods may provide a more comprehensive assessment of the technology for use.

Journalism Ethics considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.