Skip to main content
NCBI home page
Elsevier - PMC COVID-19 Collection logoLink to Elsevier - PMC COVID-19 Collection
. 2023 Jun 24:100050. Online ahead of print. doi: 10.1016/j.hfh.2023.100050

Improving Quality of Care for Patients Receiving Care through Telehealth in the Time of COVID-19 Global Pandemic and Beyond: HCI-based Leading Indicators for Virtual Visits

Yalda Khashe a, Maryam Tabibzadeh b,⁎, Najmedin Meshkati c
PMCID: PMC10290161  PMID: 38620160

Abstract

The COVID-19 pandemic expedited the growing rate of reliance on telehealth, as it provided a safer option for patients to seek the care they need and avoid potential negative consequences of being exposed to the COVID-19 virus. The aim of this study is to develop a series of Human-Computer Interaction (HCI)-based leading indicators to proactively analyze and evaluate the user interface in telehealth and virtual visits. Building on Nielsen's usability heuristics and mapping them to the six aspects of quality of care introduced by the Institute of Medicine, we identified the design features that had the highest impact on the quality of care and developed a list of leading indicators for each feature. Further, we developed corresponding checklists for each leading indicator to evaluate the features of the user. Beyond the benefits of telehealth for both patients and healthcare providers during atypical circumstances, the changes prompted by the COVID-19 public health emergency have possibly altered the position of telehealth to the point that communicating through video and audio has become the new normal. Therefore, the importance of designing an interface to facilitate user interaction with the system and consequently with one another is of utmost importance.

Keywords: human-computer interaction, interface design, usability testing and evaluation, digital health, patient safety

1. Introduction

The move towards the digitization of the healthcare system had gained a lot of traction in recent years with the advancement of technology and widespread use of digital systems. However, the COVID-19 global pandemic with the stay-at home orders and social-distancing guidelines expedited the migration of the in-person health delivery systems to virtual care. Remote access to healthcare services provided a safer option for patients to seek the care they need and avoid potential negative consequences of delayed care by limiting the potential exposures to the COVID-19 virus. According to the Center for Disease Control and Prevention (CDC), these systems also “reduced the strain on healthcare systems by minimizing the surge of patient demand on facilities and reduce the use of [Personal Protective Equipment] PPE by healthcare providers” (Centers for Disease Control and Prevention, 2020). A report from the U.S. Department of Health and Human Services found that in April 2020, 43.5% of Medicare primary care visits were provided through telehealth compared with less than 0.1% in February before the start of the COVID-19 public health emergency (United States Department of Health and Human Services, 2020). Studies also show that telehealth is here to stay (Telehealth is here to stay, 2021). Although the peak in telehealth usage declined from April to June 2020, it has since stabilized at a level 38 times higher than before the pandemic (Bestsennyy, Gilbert, Harris, & Rost, 2021).

Telehealth is defined as providing long-distance care and education facilitated by the use of electronic information and telecommunications technologies, such as videoconferencing and wireless communication (United States Department of Health and Human Services, 2020). A 2019 consumer survey report found that 66% of consumers are interested and willing to use telehealth (American Well, 2019). Beyond the benefits of telehealth for both patients and healthcare providers during atypical circumstances, the changes prompted by the COVID-19 public health emergency have possibly altered the position of telehealth in both the national and global healthcare system to the point that communicating through video and audio has become the new normal for patients and healthcare providers (Mann et al., 2020). Studies show that with the acceleration of adoption of telehealth by patients and healthcare providers, the digital health market could potentially rise up to $250 billion of current United States healthcare spend (Bestsennyy, Gilbert, Harris, & Rost, 2020).

Healthcare organizations are complex and inherently prone to risk of failure. These systems depend on the latest technologies to survive and function properly. Therefore, introducing new technologies to such an organization is inevitable (Khashe and Meshkati, 2020). Studies show that the installation of a new technology always involves some changes to the organization and its members (Majchrzak and Meshkati, 2001). An essential, but often overlooked component of implementing automated control systems such as PTC is the incorporation of human elements (Khashe and Meshkati, 2015).

Patient satisfaction has always been a growing concern of healthcare providers (Kruse et al., 2017). However, the more significant aspect of digital health is the quality of care that the patient receives virtually and the impact of technology on patient safety, as distance could hamper the communication between the patient and healthcare provider. The design of the telehealth system, and specifically the interface that connects the patient with the healthcare provider plays a crucial role in the quality and reliability of the care that patient receives. Telehealth, as a technology-mediated system, can be explored through the lens of Human-Computer Interaction (HCI) to better understand how users, in this case patients and health care providers, interact with the technology.

In this study, we evaluate the factors that could potentially impact system reliability and the quality of care offered virtually, with a main focus on HCI application in telehealth and virtual visits. We develop a set of context-specific leading indicators to create a proactive approach to analyze and assess the interface that is utilized in virtual visits to connect the two user populations, i.e. the patient and the healthcare provider.

Leading indicators are “proactive, preventive and predictive measures” (Occupational Safety and Health Administration, 2019, p.2; National Safety Council, 2014, p.3) that evaluate the performance and safety of operations in complex technological systems. Leading indicators contribute to the identification of sources of failure in a system before they occur. To the best of our knowledge by the time of this study the concept of leading indicators has not been applied to telehealth systems and HCI in the context of virtual visits.

In section 2, we discuss the growing field of telehealth in the United States and the critical role of HCI in the reliability of the virtual health system and the quality of care. In section 3, we introduce the concept of leading indicators and describe their implication in different domains. In section 4, we develop a set of HCI-based leading indicators to enable the designers to proactively analyze and assess the user interface in the context of virtual visits.

2. Telemedicine and Human-Computer Interaction

The COVID-19 global pandemic expedited the use of technology-mediated treatments, such as telehealth and virtual visits. Healthcare providers rely on the use of technology to monitor patients from their homes. National and global health guidelines that outline social distancing as one of the major preventive actions in the spread of this deadly disease have only emphasized the use of technology in healthcare. Telehealth is the use of information and communication technology to provide clinical treatments over distances (Jalil, Myers, Atkinson, Soden, 2019). According to the World Health Organization (WHO), telehealth enables all healthcare professionals to deliver care where distance is a critical factor using information and communication technologies (World Health Organization, 2010). Studies also show that the adoption of telehealth during a public health emergency reduces the time required to get a diagnosis and initiate treatment, facilitates follow-up with patients who are quarantined at home, reduces movement of people, minimizes the risk of intra-hospital infection by reducing the movement of people, and assists in informing the general public (Bokolo, 2020).

While telehealth provides a unique opportunity to continue care, especially during public health emergency, it could be potentially challenging for users who are not comfortable with technology or have difficulties with conducting virtual visits in lieu of in-person visits (Centers for Disease Control and Prevention, 2020).

Also, some patients were concerned about the lack of physical presence of a healthcare provider (Rahimpour, Lovell, Celler, & McCormick, 2007) and required trainings associated with the use of new technology and the telecommunication software (Leite, Hodgkinson & Gruber, 2020). To provide a seamless patient experience, the entire care team, including doctors, nurses, physician assistants, and any member, who might interact with the patient through the telehealth system, needs to undergo comprehensive training (Doshi et al., 2020). Thus, telehealth systems should not only be functional and reliable, but also intuitive and user-friendly (Yellowlees, 2005; Klaassen, van Beijnum and Hermens, 2016). Therefore, an important factor in effectiveness of telehealth systems is usability, or in other words the extent to which users can easily learn to work with the system reliably and safely (Charness et al., 2013).

Patient satisfaction has always been a growing concern of healthcare providers (Kruse et al., 2017). However, the more significant aspect of digital health is the quality of care that the patient receives virtually and the impact of technology on patient safety. Although quality and reliability of the software being used to connect the patients with their healthcare provider are important in the quality of the care they receive, the design of the interface being used is critical as physicians and other clinicians may not detect subtle cues that they could detect in person, while delivering care at distance (Agboola, Bates, & Kvedar, 2016). Software reliability by itself is not a comprehensive measure of telehealth performance as a socio-technical system and cannot foresee the impact of digitization upon already established work procedures and safety measures (Sujan, Scott & Cresswell, 2019). To enable the healthcare provider to provide care at distance, both patients and healthcare provider need to become viable entities of the telehealth system. (Mort, May & Williams, 2016). Failures in human components of the system could potentially lead to more harmful consequences than failures in technological components of the system (Charness et al., 2013).

Human Factors and Ergonomics (HFE) is a multidisciplinary scientific discipline concerned with the systematic study of human performance, their abilities, characteristics, and limitations and their application to the design of equipment they use, environments in which they function, and jobs they perform (Human Factors and Ergonomics Society, 2020). Human factors in operationalizing telehealth was among the top 10 patients safety concerns in the 2022 list published by the ECRI (2022).

HCI is a branch of HFE influenced by the information science and technology, focused on the interaction between the human (user) and the computer (technology) subsystems (Dix, Finlay, Abowd, & Beale, 2004). Telehealth, as a technology-mediated system, can be explored through the lens of HCI to better understand how users, in this case patients and healthcare providers, interact with the technology. While in HCI the designer usually focuses on the user-centered design of an interface that facilitates the interaction with the technical part of the system, in this study we look at the design of the telecommunication technologies, specifically videoconferencing technologies, that connects the two user populations together. In other words, we look at telehealth as the interaction mediator between the patient and the healthcare provider, and focus on the usability and design of the interface that facilitates seamless interaction of the two agents with the technology and ultimately each other (Figure 1 ).

Figure 1.

Figure 1

Open in a new tab

Patient-Healthcare Provider Interaction mediated by telehealth.

In this study, we evaluate the factors that could potentially impact system reliability and the quality of care offered virtually, with a main focus on HCI application in telehealth. To achieve this goal, we developed a set of HCI-based leading indicators that healthcare providers can use to proactively analyze and assess the user interface in the context of virtual visits.

3. Leading Indicators and Their Implications in Different Industries

The concept of leading and lagging indicators have been a point of discussion in different safety-critical industries. The role of leading indicators, in contrast with lagging indicators, is to identify sources of failure before they occur. According to a recent document entitled “Using Leading Indicators to Improve Safety and Health Outcomes”, which is published by the Occupational Safety and Health Administration (OSHA), “leading indicators are proactive, preventive, and predictive measures that provide information about the effective performance of your safety and health activities. They measure events leading up to injuries, illnesses, and other incidents and reveal potential problems in your safety and health program” (OSHA, 2019, p.2). The National Safety Council (NSC) has a similar definition for leading indicators. According to the NSC (2014, p.3) expert panel, leading indicators are: “Proactive, preventative, and predictive measures that monitor and provide current information about the effective performance, activities, and processes of an EHS [Environment, Health and Safety] management system that can drive the identification and elimination or control of risks in the workplace that can cause incidents and injuries.”

In contrast, lagging indicators capture events that have already occurred and measure their different aspects such as the frequency of their occurrence. The number or rate of injuries and fatalities are two examples of lagging indicators. Therefore, while lagging indicators can measure past incidents and alert someone about an occurred failure or existence of a hazard, leading indicators are pre-incident measurements that provide “a check of system functioning” and enable taking preventive actions to address that failure or hazard before turning into an incident.

A good program usually utilizes both lagging and leading indicators, the first to measure some level of effectiveness and the latter to drive change and act proactively. For instance, the amount of time to respond to a safety hazard report can be a leading indicator (OSHA, 2019). Any decrease in the response time can be an indication of an increase in safety awareness and managers’ commitment to workplace safety (Ibid, 2019). On the other hand, if this response time increases that can indicate management's lack of concern, which could mean that it is likely that hazards remain uncontrolled and eventually lead to incidents. Moreover, that can discourage workers from reporting hazards if they feel that management is not responsive to their concerns, which can impact employees’ morale and have further negative consequences in the workplace (Ibid, 2019).

Not only does utilizing leading indicators contribute to improving safety and health of operations, but also it can enhance efficiency and productivity. For example, identifying and addressing hazards can contribute to cost savings; e.g. repair and production costs, and employees’ compensation costs, that are common with the occurrence of incidents.

Our extensive literature review shows that the oil and gas industry (both upstream, midstream and downstream), nuclear power, electric utility, and aviation sector are among those safety-sensitive industries that have applied the concept of leading indicators to a variety of their safety processes. Many federal and safety agencies and entities have strongly recommended using lagging and leading safety indicators to improve safety and reduce accident potential in the upstream and downstream facilities associated with oil and gas industries. For instance, the National Academy of Engineering/National Research Council (NAE/NRC 2011) investigating the BP Deepwater Horizon offshore drilling rig accident explicitly recommended that, “Industry, BSEE [Bureau of Safety and Environmental Enforcement of the U.S. Department of the Interior], and other regulators should foster an effective safety culture through consistent training, adherence to principles of human factors, system safety, and continued measurement through leading indicators” (emphasis added, Recommendations 5.5 and 6.25, p. 82 and 96). The U.S. Chemical and Hazard Investigation Board's (CSB) 2007 BP Texas City investigation report describes the importance of analyzing leading and lagging indicators (CSB, 2007) and has reiterated it in its comprehensive final reports of investigations of Tesoro Anacortes Refinery (CSB, 2014) and Chevron Richmond Refinery (CSB, 2015) accidents.

The American Petroleum Institute (API, 2010) and the International Association of Oil and Gas Producers (IOGP, 2011) have stated the importance of leading indicators and introduced some of those indicators mainly from the upstream perspective. The American Bureau of Shipping (ABS, 2014) has highlighted the connection between safety culture and leading indicators of safety and developed associated leading indicators. From the downstream side of the oil and gas industry, the Center for Chemical Process Safety (CCPS, 2011) has developed guidelines for developing and monitoring context-specific lagging and leading indicators. In addition to federal or industry affiliated publications, there have been other studies to apply the concept of leading indicators to the safety operations in the oil and gas industry; e.g. Jabbari et al., 2015; Johnsen et al., 2012 and 2013; Grabowski et al., 2007.

The nuclear power industry is another safety-critical industry that has highlighted the important role of leading indicators in the safety of its operations and developed leading indicators as well as guidelines on how to utilize those indicators to monitor process safety. The International Atomic Energy Agency (IAEA, 2000) considered safety culture-related factors, such attitudes towards procedures, policies and rules, as “strategic indicators” that should always be monitored. Moreover, the Defense Nuclear Facilities Safety Board (DNFSB) has extensively studied application and efficacy of leading safety indicators for improving safety of its oversight nuclear facilities (Robinson, 2012). And it considers leading indicators to “predict the likelihood of an accident before it occurs; prevent accidents, and support productivity and quality” (Winokur and Minnema 2010, p. 6). Nevertheless, it points out that leading indicators should not be viewed necessarily as predictors of accidents; rather, they are “identifiers of accident-prone situations”. Mengolini and Debarberis (2008) is another study that uses leading indicators to develop an effectiveness evaluation methodology to enhance organizational culture in hazardous installations in the nuclear industry.

Furthermore, there are developed studies in the electric utility context; e.g. Sedgwick and Wands, 2012, and the aviation industry; e.g. Edkins, 1998, that has introduced the use of leading indicators to improve the safety of operations.

The healthcare industry, compared to the above-mentioned safety-sensitive industries, is fairly new in utilizing leading indicators to improve the safety and quality of its operations. Most research in patient safety tends to focus on reactive methods towards safety management. Among developed proactive methodologies to improve patient safety, some recent studies have proposed the use of leading indicators to create that proactive characteristic. For instance, Wormnaes (2015) introduces the concept of leading indicators as a means for real time monitoring of risk in healthcare organizations. Raben (2017) and Raben et al. (2018) propose a systematic framework to identify leading indicators in the healthcare industry. Tabibzadeh and Jahangiri (2018 and 2020) proposed a proactive risk assessment framework, through the introduction of a series of context-specific leading indicators, to enhance patient safety in operating rooms. And, Almost et al. (2018 and 2019) introduces few leading indicators for occupational health and safety management systems in healthcare, in Ontario, Canada.

Although there are studies on the application of leading indicators in improving patient safety in healthcare, to best of our knowledge, it has not been applied to telehealth. Furthermore, we have taken our emphasis beyond patient safety and have focused on improving quality of care for patients receiving care through telehealth.

3.1. Characteristics of Leading Indicators

Different organizations have discussed characteristics of leading indicators (API, 2010; IAEA, 2000; NSC, 2014; OSHA, 2019). Table 1 summarizes some of those main characteristics.

Table 1.

Characteristics of leading indicators proposed by various organizations.

Organization Characteristics of Leading Indicators
API (2010, p.16-17 and 19) • Reliable
• Repeatable
• Consistent
• Independent of outside influences
• Relevant
• Comparable		 • Meaningful
• Appropriate for the intended audience
• Timely
• Easy to use
• Auditable
IAEA (2000, p.23) • Direct relationship between the indicator and safety
• Availability of required data
• Quantitatively measurable
• Unambiguous
• Significant enough
• Not prone to get manipulated
• Form a manageable set		 • Meaningful
• Applied in regular operational activities
• Get validated easily
• Connected to the cause of a failure
• Their quality can be controlled and verified
• Having a basis upon which local actions can be taken
NSC (2014, p.3) • Actionable
• Achievable
• Meaningful
• Transparent		 • Actionable
• Achievable
• Useful
• Timely
OSHA (2019, p.2,3) • Specific
• Measurable
• Accountable		 • Reasonable
• Timely
Open in a new tab

4. Development of HCI-based Leading Indicators to Improve Quality of Care in Telehealth

4.1. Quality of Care and its Aims/Aspects

The quality of care in the healthcare industry can be assessed from different perspectives. What we will use as a point of reference in this study is what the Institute of Medicine (IOM), in its published report entitled “Crossing the Quality Chasm: A New Health System for the 21st Century”, has stated as the six aims/aspects for quality of care: safe, effective, efficient, timely, patient-centered, and equitable (IOM, 2001). The definition of each aspect with minor adjustments for the scope of our study; telehealth and virtual visits, is as follows:

  • •

    Safety: avoiding harm to patients due to receiving care through telehealth

  • •

    Effectiveness: providing services based on scientific knowledge to all who could benefit and refraining from providing services to those not likely to benefit (avoiding underuse and overuse, respectively).

  • •

    Efficiency: avoiding waste (time due to extra steps, not understanding what is going on, process methods, unproductive visits, which requires the need for more visits)

  • •

    Timeliness: reducing waits and sometimes harmful delays for both those who receive and those who give care.

  • •

    Patient-centered design: providing care that is respectful of and responsive to individual patient preferences, needs, and values and ensuring that patient values guide all clinical decisions (a care that is customizable based on patient preferences and needs)

  • •

    Equitable: providing care that does not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socio-economic status.

4.2. Usability and Heuristic Evaluation

Usability is defined as “a quality attribute that assesses how easy it is for the users to use the interface to interact with the system” (Nielsen, 2012). There are different methods to evaluate the usability of an interface in HCI. Heuristic evaluation is an approach to systematically assess a user interface design, based on user's needs and requirements, for usability (Nielsen and Molich, 1990). In this method the knowledge of an average user is applied, often guided by heuristics, to identify usability problems of an interface (Sharp, Rogers, & Preece, 2004). The goal of heuristic evaluation is to find and consequently resolve the usability issues with the design of a user interface as part of an iterative design process (Nielsen, 1994).

The heuristic evaluation has been conducted in number of studies in telemedicine to evaluate the usability of the system (Narasimha et al., 2016; Parmanto et al., 2016; Lilholt, Jensen, & Hejlesen, 2015; Stellefson, Chaney, and Chaney, 2014; Zhang and Walji, 2011; Tang et al., 2006; Kushniruk and Patel, 2004). However, to the best of our knowledge it has not been applied to the context of virtual visits in telehealth. Nielsen's heuristics has been widely used in HCI. It has also been adopted in the field of healthcare to evaluate medical devices (Tang et al., 2006). In this study, we are using Nielsen's 10 usability heuristics for user interface design (Nielsen, 1995), which is explained in detail in section 4.3.

4.3. Mapping Nielsen's Heuristics to Quality of Care Aims/Aspects

We used Nielsen's 10 usability heuristics for user interface design (Nielsen, 1995) and mapped them to the six aspects of quality of care (IOM, 2001) to evaluate the teleconferencing interfaces generally used for virtual visits. Different applications are used by various healthcare providers including Apple FaceTime, Facebook Messenger video chat, Google Hangouts video, Zoom, or Skype (U.S. Department of Health & Human Services, 2020). Then we ranked each heuristic on a scale of 1 to 5 based on the severity of its impact on the six aspects of quality of care, from two separate perspectives of the patient and the healthcare provider (Table 2 ). It should be noted that two heuristics, error prevention and recovery from error, were combined into one category since we are not evaluating a specific interface, rather looking at the virtual visit as a general application of digital health. In this study, we focused on those design evaluation heuristics that their impact score for at least one of the six aspects of quality of care is 4 and above.

Table 2.

Nielsen's Usability Heuristics and Quality of Care Impact Matrix.

Quality of Care Aims Usability Heuristics Safety Effectiveness Efficiency Timeliness Patient- centered design Equity
1. Visibility of the system P:2 P:1 P:4 P:2 P:2 P:1
H:2 H:4 H:2 H:2 H:1 H:1
2. Match between the system and the real world P:4 P:1 P:3 P:2 P:1 P:4
H:2 H:1 H:2 H:2 H:1 H:1
3. User control and freedom P:1 P:1 P:4 P:3 P:1 P:1
H:3 H:1 H:3 H:3 H:1 H:1
4. Consistency P:2 P:1 P:2 P:2 P:2 P:1
H:2 H:1 H:2 H:2 H:1 H:1
5. Error prevention and recovery P:5 P:3 P:5 P:3 P:2 P:3
H:5 H:3 H:5 H:3 H:1 H:1
6. Recognition rather than recall P:2 P:1 P:3 P:1 P:2 P:2
H:2 H:2 H:3 H:1 H:1 H:1
7. Flexibility and efficiency of use P:1 P:1 P:2 P:2 P:2 P:4
H:1 H:1 H:4 H:2 H:1 H:1
8. Aesthetic and minimalist design P:2 P:1 P:3 P:1 P:1 P:1
H:2 H:1 H:3 H:1 H:1 H:1
9. Help and documentation P:3 P:1 P:4 P:2 P:1 P:4
H:3 H:2 H:4 H:2 H:1 H:1
Open in a new tab

P: Patient, H: Healthcare Provider

In the next section we define each heuristic and then further elaborate on the associated quality of care aspects that scored 4 and above in our study:

  • (1)

    Visibility of the system: Users receive information about the status of the system and receive timely feedback

Effectiveness: If the healthcare provider is not informed about the status of the system and does not receive appropriate feedback within a reasonable time period, it might affect the quality of the information that they ask and receive from the patient and in the absence of physical cues, they might not be able to form the accurate diagnostic and provide the necessary care.

Efficiency: If the patient is not informed about the status of the system and does not receive the appropriate feedback in a timely manner, it might result in confusion and consequently requiring more time to successfully interact with the system.

  • (1)

    Match between the system and the real world: Designing an interface that matches the user's mental model and resembles the in-person interaction using simple languages and concepts from the real world.

Safety: mismatch between users’ expectation of the virtual visit and their previous experiences of in-person visits could inadvertently affect their ability to communicate with the healthcare provider. In virtual visits in lieu of physical interaction, the healthcare provider relies solely on the information provided by the patients and their caregivers and any potential miscommunication could result in diagnostic errors.

Equity: Different user populations might have a different mental model due to personal characteristics such as gender, ethnicity, geographic location, and socio-economic status. While a designed virtual system has to accommodate people with different backgrounds and mental models, discriminatory experiences due to those differences have to be avoided. Hence the HCI design should provide an equitable experience for patients with all backgrounds.

  • (1)

    User control and freedom: Users are able to navigate through steps freely to complete a task, including exit and save

Efficiency: If users are not able to backtrack steps or move between menus and options, that will contribute to additional processing time.

  • (1)

    Consistency: Consistent use of design elements, words, situations, and actions throughout the system

  • (2)

    Error prevention and recovery: How easy it is for users to make an error while interacting with the system and how they can recognize, diagnose and recover from errors

Safety: In this study, we are focusing on virtual visits. Designing an interface that minimizes the opportunities for users to make an error and the ability of the users, either patient or the healthcare provider, to recognize, diagnose and recover from errors are considerably more crucial in the applications that require disclosing, recording or storing patient-related health information; e.g. Electronic Health Records (EHR).

Efficiency: Patients rely on digital health systems to set up an appointment and virtually visit their healthcare providers. Any issue that affects their ability to make an appointment or disrupts their visits could have harmful consequences. Considering that users have various backgrounds and different levels of proficiency when interacting with a virtual system, it is important that they are able to recognize, diagnose and recover from system errors, and successfully complete their visits. Healthcare providers are mostly relying on the system to meet and interact with their patients. In the absence of physical interactions, the healthcare providers are relying on visual and auditory cues to communicate with the patient and make accurate diagnosis. Any potential system error that affects their ability to successfully interact with the patient could harm the patient.

Efficiency: Users’ inability to recognize, diagnose and recover from errors contributes to increasing the time and the effort they need to interact with the system.

  • (1)

    Recognition rather than recall: Minimize the user's memory load by making objects, actions, and options visible

  • (2)

    Flexibility and efficiency of use: Accelerators (i.e. shortcuts) have been provided to allow more experienced users to carry out tasks more quickly

Efficiency: Healthcare providers are more exposed to the digital health system than patients. Therefore, they become more experienced in interacting with the interface. If the interface is designed in a more flexible manner, it will allow more experienced users to carry out the tasks more efficiently.

Equity: The underlying assumption in the HCI is that the interface is designed in a way to accommodate all user populations, hence simple enough so that a novice user is able to interact with the system, and the flexibility heuristic usually addresses the system flexibility to allow a more experienced user to carry out tasks more quickly. However, we want to emphasize the importance of the flexibility of digital health systems to accommodate inexperienced users; especially underrepresented users who have limited exposure to digital systems and how their interaction with the system could affect the quality of care they receive.

  • (1)

    Aesthetic and minimalist design: Simple design that avoids displaying unnecessary or irrelevant information.

  • (2)

    Help and documentation: An ideal design is the one that can be used without a manual. However, that might not always be possible. Alternatively, a system is desired where help information can be easily accessed and followed by all user populations.

Efficiency: Users’ ability to access the information that they need to successfully complete their tasks affects the time and effort required to interact with the system.

Equity: The digital health system should be designed so that users with any background are able to use the system. Considering that most patients are new to digital health and some might not be proficient in using digital systems, access to the information and help documentation plays a significant role in their experience with the system.

4.4. Development of HCI-based Leading Indicators for the Most Impacted Aspects of Quality of Care

Based on the above-described analysis, the following six design evaluation heuristics are identified as the most influential on the quality of care: visibility of the system, match between the system and the real world, user control and freedom, error prevention and recovery, flexibility and efficiency of use, and help and documentation.

In this section, we developed a series of HCI-based leading indicators for each of the six above-mentioned heuristics. The development of these leading indicators was accomplished in three steps. We first developed the main categories of leading indicators for each of the heuristics. Each category was then broken down into sub-categories for further elaboration. For each sub-category we devised a checklist that can be used to evaluate the features of the user interface in the context of virtual visits. The developed leading indicators and corresponding checklist are presented in Table 3 .

Table 3.

HCI-based Leading Indicators for the Most Impacted Aspects of Quality of Care.

Leading Indicators
Visibility of the system • Designing an Inherently Visible System
• Visual System Feedback		 • Does the system provide visibility?
• Are there context labels, menu maps, place markers and consistent icon design scheme and stylistic treatment to facilitate users’ navigation across the system?
• Does the system indicate the method by which it saves information?
• Is there some form of system feedback for every user's action?
 • The user can recognize the state of the system at any point of time and knows how to proceed.
• The user is kept informed of the system progress in case of any observable delays.
• Every display starts with a title or headline to describe its content.
• In a multi-page screen, not only each page is labeled properly, but also its relation with other pages is shown clearly.
• There is a consistent schematic design and visual cues to facilitate users’ easy transition across the system.
• The system indicates whether it auto-saves information, it requires a deliberate action to save, or it is a combination of both.
• After an action or a group of actions is completed, the system provides feedback to indicate that the next action or group of actions can be started.
• Visual feedback is provided by the system when objects are selected or moved.
• Feedback response time is appropriate to the task.
Leading Indicators
Match between the system and the real world • Match between system design and users’ mental model
 • Does system design leverage familiarity with real-world objects and activities?
• Does the system speak the users’ language in familiar, rather than system-oriented terms?		 • Icons are concrete and familiar.
• Related items appear on the same display.
• Used colors correspond to common expectations about color codes.
• Menu choices are grouped and ordered in the most logical way for the intended user population.
• System language is task-oriented and familiar to users; e.g. according to their expectations and mental model.
• The words in the prompt message are consistent with the action it requires.
User control and freedom • System support for undo and redo
• Users’ freedom to select, sequence, and finalize tasks		 • Can users easily cancel or reverse their actions?
• Can users easily and efficiently select, sequence, and complete tasks?		 • System allows users to reverse their actions.
• Multiple undos are permitted when the system allows users to reverse their actions.
• An undo function is available both at a level of single entry and a group of actions.
• Users can cancel out of operations in progress.
• Users can manage their own default settings.
• System allows users to assess relationships of displayed information and adjust any inaccurately placed information
• System generates prompts to notify users about consequences of their actions.
• System requires confirmation from the user before processing a completed task.
• System supports save, exit, and return option.
Flexibility and efficiency of use • Flexibility of use • Does the system provide flexibility of use; e.g. alternative means of access and operation, for both experienced and inexperienced users? • Error messages are designed with multiple levels of details to support both novice and expert users.
• Inexperienced users can enter the most common form of commands while expert users can modify the commands.
• Expert users can utilize shortcuts to bypass nested dialog boxes and menus.
• System provides function keys for frequent commands.
Leading Indicators
Error prevention and recovery • Designing a system with error prevention in mind
• Warning Management
• Error Monitoring
• Continual Improvement
• Training
 • Does the system prevent users from making errors?
• Does the system warn users if they are about to make a potential error?
• Do error messages help users recognize an error and recover from it?
• Do error messages inform users of the severity of and consequences of their actions?
• Are system generated errors tracked and monitored?
• Are user errors tracked and monitored?
• Is the system continuously evaluated and updated?
• Do healthcare providers go through sufficient training before using the system to interact with patients?		 • Provide default values for entry fields or suggest required information when possible.
• Critical information has to be proactively tagged to avoid users’ errors.
• The system generates prompts to notify users about consequences of their actions.
• Error messages are expressed in a clear and simple language to describe the nature of the problem and suggest a way to solve it.
• Information is presented in a simple and well-organized way to enable users to easily identify inaccurate or inappropriate items.
• Error messages are designed with multiple levels of details to support both novice and expert users.
• Rate and the nature of system generated errors are captured over time and reported.
• Rate and the nature of user errors are captured over time and reported.
• There are established goals to guide and measures to track the continual improvement.
Help and documentation • Help options and documents
 • Are help options and documents visible, consistent with the interface design and easily accessible?
• Are provided help documents current and accurate?		 • The help function is visible.
• Accessing the help system and returning from it is easy.
• Users are able to return and continue work after accessing help options.
• There are navigation and completion instructions to support data entry screens and dialog boxes.
• Memory aids are embedded in the design of system commands.
• There is consistency between the interfaces (navigation, presentation and conversation) of the help system and the application it supports.
• Help documentations provide multiple levels of context-specific detail tailored for different users.
• Help documents provide accurate, complete, and understandable information.
Open in a new tab

Our extensive literature review of existing leading indicators in various safety-sensitive industries was used as the basis for the development of the main categories of leading indicators. The development of leading indicators sub-categories and the checklists were inspired by previous works in the field of usability testing by Pierotti (2014) and Lowry et al. (2012).

5. Concluding Remarks and Future Research

The expanding application of telehealth in the time of a public health emergency has been put to test with the recent COVID-19 crisis. Although receiving care virtually had gained traction in recent years with the advancement of technology and widespread use of digital systems, the COVID-19 pandemic has expedited the growing rate of reliance on telehealth, as it provides a safer option for patients to seek the care they need and avoid potential negative consequences of being exposed to the COVID-19 virus.

An important factor in effectiveness of telehealth systems is usability, or in other words the extent to which users can easily learn to work with the system reliably and safely. The design of the user interface is especially important, as it connects the patient with the healthcare provider and its performance can affect the quality of care received by the patient. Telehealth, as a technology-mediated system, can be explored through the lens of HCI to better understand how users, in this case patients and health care providers, interact with the technology.

In this study, we developed a series of HCI-based leading indicators to proactively analyze and evaluate the user interface in virtual visits. Which connects the two user populations, i.e. the patient and the healthcare provider. We used Nielsen's 10 usability heuristics for user interface design (Nielsen, 1995) and mapped them to the six aspects of quality of care (IOM, 2001) to evaluate the teleconferencing interfaces generally used for virtual visits. We identified the design features that had the highest impact on the quality of care.  Error prevention and recovery, with four severity rankings of five, was identified as the most influential design feature, followed by help and documentation, visibility of the system, match between the system and the real work, flexibility and efficiency of use, and user control and freedom.

The aforementioned leading indicators were developed for each of the six identified design features. Further we developed corresponding checklists for each leading indicator that can be used to evaluate the features of the user interface in the context of virtual visits.

One of the directions for future research is to create a review system using our developed leading indicators and checklists to evaluate the design of the user interface in telehealth and expanding its application to the other areas of digital healthcare. In the method described in this paper, the system can be evaluated by a group of experts even before implantation, hence minimizing the challenges of user testing and consequent cost of design changes. However, the devised list of leading indicators and accompanied checklists can be used to monitor and improve the system after implementation.

Beyond the benefits of telehealth for both patients and healthcare providers during atypical circumstances, the changes prompted by the COVID-19 public health emergency have possibly altered the position of telehealth in both the national and global healthcare system to the point that communicating through video and audio will become the new normal for patients and healthcare providers (Mann et al., 2020). Therefore, the importance of designing an interface to facilitate user interaction with the system and consequently with one another is of utmost importance.

Uncited References

United States Department of Health and Human Services 2020, United States Department of Health and Human Services 2020

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

  1. Agboola S., Bates D., Kvedar J. Digital Health and Patient Safety. JAMA : the Journal of the American Medical Association, 2016;315(16):1697–1698. doi: 10.1001/jama.2016.2402. [DOI] [PubMed] [Google Scholar]
  2. Almost J.M., Tett L.C., VanDenKerkhof E., Paré G.C., Strahlendorf P., Noonan J., Hayes T., Van hulle H., Holden J., Silva e Silva V., Rochon A. Leading Indicators in Occupational Health and Safety Management Systems in Healthcare: A Quasi-Experimental Longitudinal Study. Journal of Occupational and Environmental Medicine. 2019;61(12):e486–e496. doi: 10.1097/JOM.0000000000001738. [DOI] [PubMed] [Google Scholar]
  3. Almost J.M., VanDenKerkhof E., Strahlendorf P., Tett L.C., Noonan J., Hayes T., Van hulle H., Adam R., Holden J., Kent-Hillis T., McDonald M., Paré G.C., Lachhar K., Silva e Silva V. A study of leading indicators for occupational health and safety management systems in healthcare. BMC Health Services Research. 2018;18:296. doi: 10.1186/s12913-018-3103-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. American Bureau of Shipping (ABS) American Bureau of Shipping; Houston, TX: 2014. Guidance notes on Safety culture and leading indicators of safety. [Google Scholar]
  5. American Petroleum Institute (API) API; Washington, DC: 2010. Process Safety Performance Indicators for the Refining and Petrochemical Industries, Recommended Practice 754. April. [Google Scholar]
  6. American Well. (2019). Telehealth Index: 2019 Consumer Survey. From, https://static.americanwell.com/app/uploads/2019/07/American-Well-Telehealth-Index-2019-Consumer-Survey-eBook2.pdf
  7. Bestsennyy, O., Gilbert, G., Harris, A., & Rost, J. (Update: 2021, July 9). Telehealth: A quarter-trillion-dollar post-COVID-19 reality? From McKinsey & Company: https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/telehealth-a-quarter-trillion-dollar-post-covid-19-reality#
  8. Bestsennyy, O., Gilbert, G., Harris, A., & Rost, J. (2020, May 29). Telehealth: A quarter-trillion-dollar post-COVID-19 reality? From McKinsey & Company: https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/telehealth-a-quarter-trillion-dollar-post-covid-19-reality#
  9. Bokolo A. Exploring the adoption of telemedicine and virtual software for care of outpatients during and after COVID-19 pandemic. Irish Journal of Medical Science. 2020:1–10. doi: 10.1007/s11845-020-02299-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Center for Chemical Process Safety (CCPS) CCPS; New York, NY: 2011. Process Safety Leading and Lagging Metrics. CCPS of the American Institute of Chemical Engineers (AIChE) January. [Google Scholar]
  11. Centers for Disease Control and Prevention. (2020, June 10). Using Telehealth to Expand Access to Essential Health Services during the COVID-19 Pandemic. From Centers for Disease Control and Prevention: https://www.cdc.gov/coronavirus/2019-ncov/hcp/telehealth.html
  12. Charness N., Fox M., Papadopoulos A., Crump C. Metrics for Assessing the Reliability of a Telemedicine Remote Monitoring System. Telemedicine Journal and e-Health. 2013;19(6):487–492. doi: 10.1089/tmj.2012.0143. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Chemical Safety and Hazard Investigation Board (CSB) (2007). Refinery Explosion and Fire: BP Texas City, March 23, 2005. Investigation Report. Washington, DC: CSB.
  14. Chemical Safety and Hazard Investigation Board (CSB) CSB; Washington, DC: 2014. Investigation Report of Catastrophic Rupture of Heat Exchanger at the Tesoro Anacortes Refinery. Report 2010-08-I-WA. [Google Scholar]
  15. Chemical Safety and Hazard Investigation Board (CSB) CSB; Washington, DC: 2015. Chevron Richmond Refinery Pipe Rupture and Fire at #4 Crude Unit. Report No. 2012-03-I-CA. [Google Scholar]
  16. Dix A., Finlay J., Abowd G., Beale R. Harlow, England: Pearson Education; 2004. Human-computer interaction (3rd ed. ed.) [Google Scholar]
  17. Doshi A., Platt Y., Dressen J., Mathews B., Siy J. Keep Calm and Log On: Telemedicine for COVID-19 Pandemic Response. Journal of Hospital Medicine. 2020;15(5):302–304. doi: 10.12788/jhm.3419. [DOI] [PubMed] [Google Scholar]
  18. ECRI . The Most Trusted Voice in Healthcare; 2022. Top 10 Patient Safety Concerns 2022.https://assets.ecri.org/PDF/Solutions/Patient-Safety-Organization/2022_Top10PatientSafetyConcerns_final2.pdf Special Report. ECRIRetrieved 2022 from. [Google Scholar]
  19. Edkins G.D. The INDICATE safety program: evaluation of a method to proactively improve airline safety performance. Safety Science. 1998;30(3):275–295. [Google Scholar]
  20. Grabowski M., Ayyalasomayajula P., Merrick J., McCafferty D. Accident precursors and safety nets: leading indicators of tanker operations safety. Marit Policy Manage. 2007;34(5):405–425. [Google Scholar]
  21. Human Factors and Ergonomics Society. (n.d.). Definitions of Human Factors and Ergonomics. Retrieved 2020 from https://www.hfes.org/resources/educational-and-professional-resources/new-item
  22. International Atomic Energy Agency (IAEA) (2000). Operational Safety Performance Indicators for Nuclear Power Plants. Vienna, Austria: IAEA.
  23. Institute of Medicine (IOM) The National Academies Press; Washington, DC: 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Committee on Quality of Health Care in America. [PubMed] [Google Scholar]
  24. International Association of Oil and Gas Producers (IOGP) (2011). Process safety - recommended practice on key performance indicators. Report No. 456
  25. Jabbari N., Ashayeri C., Meshkati N. Society of Petroleum Engineers; Garden Grove, California: 2015. Leading Safety, Health, and Environmental Indicators in Hydraulic Fracturing. Western Regional MeetingApril 27-30. [Google Scholar]
  26. Jalil S, Myers T, Atkinson I, Soden M. Complementing a Clinical Trial with Human-Computer Interaction: Patients’ User Experience with Telehealth. JMIR human factors. 2019;6(2) doi: 10.2196/humanfactors.9481. e9481-e9481. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Johnsen S.O., Okstad E., Aas A.L., Skramstad T. Proactive indicators to control risks in operations of oil and gas fields. Society of Petroleum Engineers. 2012;4(2):1–16. [Google Scholar]
  28. Johnsen S.O., Okstad E., Aas A.L., Skramstad T. Proactive indicators to control risks in operations of oil and gas fields. SPE Economics and Management. 2013;4:90–105. [Google Scholar]
  29. Khashe, Y., & Meshkati, N. (2015). Evaluation of Human-Systems Interaction of the PTC Technology in Railroad, a Preliminary Study. In IIE Annual Conference. Proceedings (p. 1888). Institute of Industrial and Systems Engineers (IISE).
  30. Khashe Y., Meshkati N. Springer; Cham: 2020. New Technology Implementation in High-Risk Organizations-The Application of HRO Principles in New Technology Implementation in Railroad Industry. July. In International Conference on Applied Human Factors and Ergonomics (pp. 120-126) [Google Scholar]
  31. Kruse C., Krowski N., Rodriguez B., Tran L., Vela J., Brooks M. Telehealth and patient satisfaction: a systematic review and narrative analysis. BMJ Open. 2017;7(8) doi: 10.1136/bmjopen-2017-016242. e016242–e016242. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Klaassen B., van Beijnum B., Hermens H. Usability in telemedicine systems—A literature survey. International Journal of Medical Informatics (Shannon, Ireland) 2016;93:57–69. doi: 10.1016/j.ijmedinf.2016.06.004. [DOI] [PubMed] [Google Scholar]
  33. Kushniruk A., Patel V. Cognitive and usability engineering methods for the evaluation of clinical information systems. Journal of Biomedical Informatics. 2004;37(1):56–76. doi: 10.1016/j.jbi.2004.01.003. [DOI] [PubMed] [Google Scholar]
  34. Leite H, Hodgkinson IR, Gruber T. New development: ‘healing at a distance’—telemedicine and COVID-19. Public Money Manag. 2020;40(6):483–485. doi: 10.1080/09540962.2020.1748855. [DOI] [Google Scholar]
  35. Lilholt P., Jensen M., Hejlesen O. Heuristic evaluation of a telehealth system from the Danish TeleCare North Trial. International Journal of Medical Informatics (Shannon, Ireland) 2015;84(5):319–326. doi: 10.1016/j.ijmedinf.2015.01.012. [DOI] [PubMed] [Google Scholar]
  36. Lowry S.Z., Quinn M.T., Ramaiah M., Schumacher R.M., Patterson E.S., North R., Abbott P. National Institute of Standards and Technology, U.S. Department of Commerce; Washington, D.C.: 2012. Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records. [Google Scholar]
  37. Majchrzak A., Meshkat N. Aligning technological and organizational change. In: Handbook of Industrial Engineering. Technology and Operations Management. 2001;36:948–974. 3rd edn., ch.Wiley-Interscience Publication. [Google Scholar]
  38. Mann D., Chen J., Chunara R., Testa P., Nov O. COVID-19 transforms health care through telemedicine: evidence from the field. Journal of the American Medical Informatics Association: JAMIA. 2020;27(7):1132–1135. doi: 10.1093/jamia/ocaa072. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Mengolini A., Debarberis L. Effectiveness evaluation methodology for safety processes to enhance organisational culture in hazardous installations. Journal of Hazardous Materials. 2008;155(1):243–252. doi: 10.1016/j.jhazmat.2007.11.078. [DOI] [PubMed] [Google Scholar]
  40. Mort M., May C., Williams T. Remote Doctors and Absent Patients: Acting at a Distance in Telemedicine? Science. Technology, & Human Values. 2016;28(2):274–295. doi: 10.1177/0162243902250907. [DOI] [Google Scholar]
  41. Narasimha S., Agnisarman S., Madathil K.C., Gramopadhye A.K., Welch B., Mcelligott J. An investigation of the usability issues of home-based video telemedicine systems with geriatric patients. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; Sage CA: Los Angeles, CA; SAGE Publications; 2016. pp. 1804–1808. VolNo. [Google Scholar]
  42. National Academy of Engineering/National Research Council (NAE/NRC) (2011). Macondo Well–Deepwater Horizon Blowout: Lessons for Improving Offshore Drilling Safety. Washington, DC: National Academy Press.
  43. National Safety Council (NSC) in the Safety+Health (the official magazine of the NSC Congress and Expo); 2014. Get Started: Selecting Leading Indicators to Help Measure Safety. February 1. [Google Scholar]
  44. Nielsen J. Elsevier Science & Technology; 1994. Usability Engineering: Usability Engineering. Usability Engineering (1st ed.) [Google Scholar]
  45. Nielsen J. 10 usability heuristics for user interface design. Nielsen Norman Group. 1995;1(1) [Google Scholar]
  46. Nielsen J. From Nielsen Norman Group; 2012. Usability 101: Introduction to Usability.https://www.nngroup.com/articles/usability-101-introduction-to-usability/ January 3. [Google Scholar]
  47. Nielsen J, Molich R. Heuristic evaluation of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; Seattle, Washington, DC; 1990. pp. 249–256. [Google Scholar]
  48. Occupational Safety and Health Administration (OSHA) U.S. Department of Labor; OSHA: 2019. Using Leading Indicators to Improve Safety and Health Outcomes. June. 3970. [Google Scholar]
  49. Parmanto B., Lewis J., Graham K., Bertolet M. Development of the Telehealth Usability Questionnaire (TUQ. International Journal of Telerehabilitation. 2016;8(1):3–10. doi: 10.5195/ijt.2016.6196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Pierotti, D. (2014). Heuristic Evaluation - A Systematic Checklist. From http://stcsig.org/usability/topics/articles/he-checklist.html
  51. Raben D.C. University of Southern Denmark; Denmark: 2017. Identification of leading indicators in healthcare processes – developing a method. Doctor of Philosophy Thesis. [Google Scholar]
  52. Raben D.C., Bogh S.B., Viskumc B., Mikkelsend K.L, Hollnagel E. Learn from what goes right: A demonstration of a new systematic method for identification of leading indicators in healthcare. Reliability Engineering and System Safety. 2018;169:187–198. [Google Scholar]
  53. Rahimpour M., Lovell N., Celler B., McCormick J. Patients’ perceptions of a home telecare system. International Journal of Medical Informatics. 2007;77(7):486–498. doi: 10.1016/j.ijmedinf.2007.10.006. [DOI] [PubMed] [Google Scholar]
  54. Robinson, J.H. (2012, July 23). Using Leading Indicators to Avoid Major Accidents. Statement by Honorable Jessie Hill Roberson, Vice Chairman of the U.S. Defense Nuclear Facilities Safety Board (DNFSB) at the U.S. Chemical Safety Board's Public Hearing: Safety Performance Indicators.
  55. Sedgwick M., Wands A. Presentation at the U.S. Chemical Safety Board (CSB) Public Hearing: Safety Performance Indicators; Washington, DC: 2012. The Implementation of Effective Key Performance Indicators to Manage Major Hazard Risks (at Scottish Power) July 23, 2012. [Google Scholar]
  56. Sharp H., Rogers Y., Preece J. Wiley; 2004. Interaction design: beyond human-computer interaction. 5th ed. [Google Scholar]
  57. Stellefson M., Chaney B., Chaney D. Heuristic Evaluation of Online COPD Respiratory Therapy and Education Video Resource Center. Telemedicine Journal and e-Health. 2014;20(10):972–976. doi: 10.1089/tmj.2014.0009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Sujan M., Scott P., Cresswell K. Digital health and patient safety: Technology is not a magic wand. Health Informatics Journal. 2019 doi: 10.1177/1460458219876183. [DOI] [PubMed] [Google Scholar]
  59. Tabibzadeh M., Jahangiri G. A Proactive Risk Assessment Methodology to Enhance Patient Safety: Reducing Wrong Site Surgery as a Preventable Medical Error. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2020;64(1):664–668. [Google Scholar]
  60. Tabibzadeh, M. and Jahangiri, G. (2018). A Proactive Risk Assessment Framework to Enhance Patient Safety in Operating Rooms. In: V. Duffy and N. J. Lightner (eds), Advances in Human Factors and Ergonomics in Healthcare and Medical Devices. Advances in Intelligent Systems and Computing, 590, 263-274, Springer, Cham.
  61. Tang Z., Johnson T., Tindall R., Zhang J. Applying Heuristic Evaluation to Improve the Usability of a Telemedicine System. Telemedicine Journal and e-Health. 2006;12(1):24–34. doi: 10.1089/tmj.2006.12.24. [DOI] [PubMed] [Google Scholar]
  62. Telehealth is here to Stay (2021). Nature Medicine, 27, 1121. https://doi.org/10.1038/s41591-021-01447-x [DOI] [PubMed]
  63. United States Department of Health and Human Services. (2020, March 30). Notification of Enforcement Discretion for Telehealth Remote Communications During the COVID-19 Nationwide Public Health Emergency. From https://www.hhs.gov/hipaa/for-professionals/special-topics/emergency-preparedness/notification-enforcement-discretion-telehealth/index.html
  64. United States Department of Health and Human Services. (2020, June 28). HHS Issues New Report Highlighting Dramatic Trends in Medicare Beneficiary Telehealth Utilization amid COVID-19. From https://www.hhs.gov/about/news/2020/07/28/hhs-issues-new-report-highlighting-dramatic-trends-in-medicare-beneficiary-telehealth-utilization-amid-covid-19.html
  65. United States Department of Health and Human Services. (2020, March 27). What is telehealth? From https://www.hhs.gov/hipaa/for-professionals/faq/3015/what-is-telehealth/index.html
  66. Winokur P.S., Minnema D. EFCOG [Energy Facility Contractors Group] Annual Executive Council Meeting; 2010. DNFSB [Defense Nuclear Facilities Safety Board] Perspective on Metrics and Safety Reform. June 24. [Google Scholar]
  67. World Health Organization . World Health Organization; 2010. Telemedicine: opportunities and developments in Member States: report on the second global survey on eHealth. [Google Scholar]
  68. Wormnaes M. University of Stavanger; Stavanger, Norway: 2015. Leading indicators for real time monitoring of risk in healthcare organizations. Master's thesis. [Google Scholar]
  69. Yellowlees P. Successfully developing a telemedicine system. Journal of Telemedicine and Telecare. 2005;11(7):331–335. doi: 10.1258/135763305774472024. [DOI] [PubMed] [Google Scholar]
  70. Zhang J., Walji M. TURF: Toward a unified framework of EHR usability. Journal of Biomedical Informatics. 2011;44(6):1056–1067. doi: 10.1016/j.jbi.2011.08.005. [DOI] [PubMed] [Google Scholar]

Articles from Human Factors in Healthcare are provided here courtesy of Elsevier

RESOURCES