Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review

Abstract

Objective: To systematically review studies reporting problems with information technology (IT) in health care and their effects on care delivery and patient outcomes.

Materials and methods: We searched bibliographic databases including Scopus, PubMed, and Science Citation Index Expanded from January 2004 to December 2015 for studies reporting problems with IT and their effects. A framework called the information value chain, which connects technology use to final outcome, was used to assess how IT problems affect user interaction, information receipt, decision-making, care processes, and patient outcomes. The review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.

Results: Of the 34 studies identified, the majority (n = 14, 41%) were analyses of incidents reported from 6 countries. There were 7 descriptive studies, 9 ethnographic studies, and 4 case reports. The types of IT problems were similar to those described in earlier classifications of safety problems associated with health IT. The frequency, scale, and severity of IT problems were not adequately captured within these studies. Use errors and poor user interfaces interfered with the receipt of information and led to errors of commission when making decisions. Clinical errors involving medications were well characterized. Issues with system functionality, including poor user interfaces and fragmented displays, delayed care delivery. Issues with system access, system configuration, and software updates also delayed care. In 18 studies (53%), IT problems were linked to patient harm and death. Near-miss events were reported in 10 studies (29%).

Discussion and conclusion: The research evidence describing problems with health IT remains largely qualitative, and many opportunities remain to systematically study and quantify risks and benefits with regard to patient safety. The information value chain, when used in conjunction with existing classifications for health IT safety problems, can enhance measurement and should facilitate identification of the most significant risks to patient safety.

INTRODUCTION

The widespread adoption of information technology (IT) brings many potential benefits to health care.¹ At the same time, problems with IT can disrupt the delivery of care and increase the likelihood of new, often unforeseen, errors that affect the safety and quality of clinical care and may lead to patient harm.^2–7 Our capacity to reap the benefits of IT and manage new threats is contingent upon understanding the ways in which IT problems can disrupt care delivery and pose threats to patient safety.

While previous reviews have looked at the overall effect of IT on patient outcomes,⁸ less is known about the specific nature of IT problems in clinical settings. A recent review broadly examined 5 categories of sociotechnical factors affecting safety, including people, technology tasks, organization, and environment.⁹ In this study, we sought to summarize the research literature describing the different problems known to be associated with IT in health care and to more precisely analyze their effects on care delivery and patient outcomes.

METHODS

Study identification and selection

Health IT was broadly defined as computer hardware and software used by health professionals to support patient care. We focused on studies reporting problems with IT and its effects on care delivery and patient outcomes. These effects were examined using a new framework called the information value chain, which connects the use of a technology to final outcome (Figure 1).^10,11 The chain is initiated when a user interacts with an IT system. A subset of these interactions will yield new information, only some of which then lead to changed decisions. Next, only some decisions will see changes in the care process, and only some process changes will impact patient outcome. Using this framework, we sought to identify the effects of IT problems on each stage of this chain, from user interaction to clinical outcome.

Fig. 1.

The information value chain connects use of a technology to final outcome.^10,11 We examined the effects of IT problems on user interaction and information received, as well as effects on decision-making, care process, and patient outcomes. An existing classification was used to categorize IT problems, information received, and contributing factors.¹²

We searched the bibliographic databases Scopus, PubMed, and Science Citation Index Expanded from January 2004 to December 2015. The search query used was (“health information technology” OR HIT OR “health IT” OR “electronic health record” OR “electronic medical record” OR “decision support” OR CPOE OR “technology induced” OR “computer related”) AND (“medication error*” OR error* OR incident* OR “unintended adverse consequence*” OR consequence* OR “incident report*” OR “patient outcome”) AND (data OR analysis OR qualitative OR quantitative). To be included in the review, studies needed to report problems with IT and their effects on 1 or more stages in the information value chain, ie, interaction, information received, decision-making, care process, or outcome. Study quality was assessed by examining study design, risk of bias, duration, population size, and reporting about successive stages in the information value chain. Only English-language studies were included.

The searches identified 3277 potential articles: Scopus, 1093; PubMed, 1197; and Science Citation Index Expanded, 987. In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting of systematic reviews, one reviewer first reviewed all titles and abstracts. Seventy-nine papers were selected for full review (Figure 2). Each study was assessed independently by 2 reviewers (MK and FM) against the inclusion criteria. All disagreements were resolved by consensus. After assessment, 34 studies remained.

Fig. 2.

Study identification and selection

Data extraction and analysis of the effects of IT problems

For each included study, both reviewers extracted information to develop an inventory of IT problems. Each identified problem was then labeled using an existing classification for safety problems associated with health IT to describe the nature of the problem.¹² The problems were then assigned to the different stages of the value chain, depending on whether they affected user interaction, information receipt, decision-making, care processes, or outcomes. Some of the identified problems had the broad potential to affect events at multiple stages of the chain, and these were classed as either general technical issues, covering problems in the design of software and hardware, or sociotechnical contextual variables related to human or organizational issues that influenced user interaction (contributing factors).

The next stage of analysis sought to further characterize these groupings. Four types of errors in information (information errors) were considered: wrong, missing, partial, and delayed.^13,14 These could arise from how software was used (use errors) or software and hardware issues (machine errors). Errors and delays in decision-making were similarly identified. We sought to identify omission errors (ie, when an intended action was not executed) and commission errors (ie, when an action was wrong).

Finally, observable impact on care process and outcomes was examined using a standard approach and categorized as^13–16:

1. Potential or actual harm to a patient. An IT problem led to a clinical error that reached the patient,¹⁶ eg, a patient had severe allergic reaction to prescribed medication.

2. An arrested or interrupted sequence or a near miss. An IT problem led to a clinical error that was detected before reaching the patient,^15,16 eg, a prescription in a wrong name was noticed and corrected while printing.

3. An IT problem with a noticeable consequence but no patient harm. A problem that affected care delivery but involved no harm to a patient, such as delays and rework, eg, a computer network problem resulted in delays or additional phone calls to follow up missing test results.

4. An IT problem with no noticeable consequence. A problem that did not directly affect the delivery of care, eg, an electronic backup copy of patient records was corrupted, but this was detected and the copy was not needed.

5. A hazardous event or circumstance. A problem that could potentially lead to an adverse event or a near miss, eg, prescribing software failed to display a patient’s allergy status.

A narrative synthesis then integrated findings into descriptive summaries for each stage of the value chain.

RESULTS

Descriptive analysis of all studies

We identified 34 studies describing the effects of IT problems on care delivery and patient outcomes (Table 1). The majority were analyses of incidents (n = 14, 41%; Table 2), which were reported at varying levels, from a single hospital to nationwide, in 6 countries: the United States, the United Kingdom, the Netherlands, China, Hong Kong, and Australia.^{12–15,17–26} Nine were ethnographic studies using interviews, surveys, and participant observation^27–35 and 7 were descriptive studies using existing data such as prescriptions to examine medication errors.^36–42 The remaining 4 studies were case reports.^43–46 Of the 34 studies reviewed, more than half examined computerized provider order entry (CPOE) or prescribing systems (n = 19) and 10 (29%) examined all types of health IT systems. Three out of 4 studies were undertaken in inpatient settings (n = 26, 76%) and 15 in outpatient settings.

Table 1.

Studies reporting the effects of health IT problems on care delivery and patient outcomes

Authors (year)	Study period (no. of months)	Method and design	Country	Sample (incidents)	Health IT type	Setting	Key findings
Aarts et al. (2004)43	1988–2000 (144)	Case study; longitudinal study	Netherlands		CPOE	Inpatient	Hospitalwide implementation of CPOE was halted due to unintended consequences. Software problems including poor user interfaces led to difficulties in entering orders and delayed care processes.
Ash et al. (2004)36		Survey and observ (qualitative)	United States, Netherlands, Australia	>340 h, >77 interviews	All health IT	Inpatient, outpatient	Two categories of unintended effects were identified: errors in processes of entering and retrieving information, and communication and coordination processes. Specific consequences for care delivery included more work for clinicians and communication loss. Technical problems were related to inadequate software functionality such as poor user interfaces, fragmented displays, and inflexibility of system design. System configuration issues were related to problems with decision support including reminders, alerts, and system messages. Contributing factors included poor integration with workflow, cognitive load due to interruptions. Workarounds were used to deal with many IT issues.
Ash et al. (2007)28		Survey (qualitative)	United States	176 responses	CPOE	Inpatient, outpatient	Representatives from 176 hospitals were surveyed about the extent and importance of CPOE-related unintended consequences. The survey was based on 9 major categories of unintended consequences of CPOE implementation identified by Campbell et al. (2006). All hospitals reported 8 categories of unintended consequences (ie, except category 4, problems related to paper persistence). Seventy-two percent of respondents ranked unintended consequences as moderately to very important, including these categories: (1) more/new work issues, (2) workflow issues, (3) never-ending system demands, (5) communication issues, (6) intense emotions, and (9) overdependence on technology.
Campbell et al. (2006)37	2004–2005 (9)	Observ (qualitative)	United States	324	CPOE	Inpatient	Unintended consequences associated with CPOE implementation were investigated. Technical problems were related to software functionality, which confused users. Clinical care delivery became dependent upon CPOE technology. System failure and malfunctions delayed patient care and required use of hybrid records systems. Overall, 9 major categories of unintended consequences were identified: (1) more/new work for a clinician, (2) unfavorable workflow issues, (3) never-ending system demands, (4) problems related to paper persistence, (5) untoward changes in communication pattern and practices, (6) negative emotions, (7) generation of new kinds of errors, (8) unexpected changes in the power structure, and (9) overdependence on technology.
Cheung et al. (2013)20	2010–2011 (12)	Incident analysis (qualitative)	Netherlands	668	CPOE, EHR, AutoDisp, PIS, infusion pump, eMAR	Inpatient, outpatient pharmacy	Half of the incidents were associated with use errors relating to wrong data entry. Technical problems were related to poor design of user interfaces, which led to medication errors in community pharmacies. A total of 360 incidents reached patients. Two deaths and 20 cases of serious but temporary harm were reported.
Han et al. (2005)42	2001–2003 (18)	Comparative (quantitative)	United States	1,942 patients	CPOE	Inpatient	CPOE implementation in a children's hospital and accompanying policy changes were associated with increased mortality from 2.8% to 6.6%. A poor user interface that was not adapted to local requirements led to delays in initiating treatment. The CPOE did not allow entry of orders prior to arrival of critically ill patients, delaying life-saving treatment. New workflow also caused a breakdown in doctor-nurse communication. Hospitalwide implementation over a 6-day period did not allow staff enough time to adapt to new routines and responsibilities. In parallel with CPOE implementation, changes to policies and procedures for dispensing and administering medications exacerbated treatment delays. For instance, all medications including ICU vasoactive drugs were relocated to a central pharmacy.
Hanuscak et al. (2009)32	2007 (3)	Survey (qualitative)	United States	32 responses	All health IT		Thirty-two individuals who were directly responsible for supporting and maintaining IT systems in 78 hospitals were surveyed to evaluate causes and frequency of medication errors due to downtime over a 12-month period. Technical problems were related to hardware and software issues, including malfunctioning systems, interfaces with other software components, and updates. Downtimes were also linked to use errors. Of the 39 medication errors linked to downtime, 14 reached patients.
Horsky et al. (2005)44		Case study	United States		CPOE	Inpatient	An elderly patient suffering from hypokalemia (low potassium; serum potassium was 3.1 mEq/L, creatinine 1.7) became severely hyperkalemic (serum potassium 7.8 mEq/L). Wrong, incomplete, and missing information in the hospital order entry system resulted in the patient receiving multiple doses of potassium. In total, 316 mEq potassium chloride was administered over 42 hours. Technical problems were related to software functionality, such as suboptimal screen display and lack of automated checking function. Human factors issues were linked with inadequate training and poor familiarity with the system.
Koppel et al. (2005)27	2002–2004 (24)	Survey and observ (qualitative)	United States	261 responses, 32 interviews	CPOE	Inpatient	Twenty-two CPOE-related mechanisms for medication errors were identified and broadly categorized by information errors and human-machine interface flaws. Technical problems involved software functionality and system configuration. Fragmented displays disrupted user interaction and led to errors in selecting medications. Poor displays delayed time to complete clinical tasks. Problems with process for reapproval of antibiotics led to gaps in therapy. CPOE downtime also contributed to delays in care process. Hybrid record systems were used to deliver care during downtimes.
Landman et al. (2013)46		Case study (qualitative)	United States		EHR	Inpatient	An update to Web browser software severed the link between an ED tracking board and a Web-based image viewer. Loss of this link resulted in decreased image viewer access rates for ED patients during the 10 days of the incident (2.8 views/study) compared with image review rates for a similar 10-day period preceding this event (3.8 views/study, P < .001)
Lei et al. (2013)25	2001–2012 (142)	Incident analysis (qualitative)	China	116	All health IT	Inpatient, outpatient, general practice	In total, 66% of downtime events were associated with technical problems, such as hardware and software malfunctions; 36% of events affected more than 100 individuals, and of these, 9 events affected over 1000 individuals. In all, 109 events impacted care delivery. In 21 events, patients were forced to seek care in other hospitals. One death was reportedly linked to downtime.
Magrabi et al. (2010)13	2003–2005 (24)	Incident analysis (qualitative)	Australia	99	All health IT	Inpatient, general practice	There were 99 incidents describing 117 problems. Overall, 55% of problems were machine-related and 45% were attributed to human-computer interaction. Delays in initiating and completing clinical tasks were a major consequence of machine-related problems (70%), whereas rework was a major consequence of human-computer interaction problems (78%). While 38% (n = 26) of the incidents were reported to have a noticeable consequence but no harm, 34% (n = 23) had no noticeable consequence.
Magrabi et al. (2012), (2011)14,50	2008–2010 (30)	Analyses of reports about equipment failure and hazards (qualitative)	United States	436	All health IT	Inpatient	A total of 436 incidents were associated with 712 problems; 96% of problems were machine related, and the remaining were problems with the human-computer interface. In all, 11% of incidents were associated with patient harm, and 4 deaths were linked to health IT problems: (1) Entry of a portable X-ray image into a PACS system under the wrong name resulted in a wrong diagnosis and subsequent intubation, which may have contributed to death. (2) A technician mistakenly entered the date of birth of a baby instead of the study date, making a chest X-ray appear older than it was. A radiologist subsequently viewed the image for peripherally inserted central catheter (PICC line) placement. Seeing that the comparison image did not have the line present, the radiologist concluded that it had been removed. Unfortunately, the line was placed too far in the infant, and the premature baby died. (3) Orders were not executed and went undetected due to inadequate separation of preoperative orders from postoperative, resulting in a “missed opportunity” to diagnose and treat a life-threatening disease, contributing to death. (4) A CPOE user interface that did not provide medication doses in milligrams was associated with administration of 3 times the maximum dose of an analgesic drug in 24 hours. This resulted in acute renal failure and death.
Magrabi et al. (2015)12	2005–2011 (74)	Analysis of all safety events reported to national IT service desk (qualitative)	England	850	All health IT	Inpatient, outpatient, long-term care, pharmacy, general practice	Of the 850 events analyzed, 68% (n = 574) described potentially hazardous circumstances, 24% (n = 205) had an observable impact on care delivery, 4% (n = 36) were a near miss, and 3% (n = 22) were associated with patient harm, including 3 deaths (0.35%). Eleven events did not have a noticeable consequence (1%) and 2 were complaints ( < 1%). Among the events, 1606 separate contributing problems were identified. Of these, 92% were predominantly associated with technical rather than human factors. There were 3 deaths: (1) A patient who was seen with another patient’s records in general practice was prescribed that patient’s medication and died later the same day from taking it. No further details were available. (2) A patient suffering from chest pain advised the receptionist in a GP surgery. The receptionist intended to alert the GP about this patient via the practice software, but sent the message to herself instead. The patient later died from a myocardial infarction. (3) An HIV test ordered during a hospital stay was not followed up after discharge. When the patient was readmitted, the admitting doctors were unable to access the HIV test result, because the test request was hidden from them. The patient developed pneumocystis pneumonia and died.
Magrabi et al. (2015)15	2012–2013 (19)	Incident analysis (qualitative)	Australia	90	All health IT	General practice	A total of 42% of the incidents had an observable impact on delivery of care but were not associated with patient harm. Of these, 6% reported potential or patient harm and 27% were a near miss. Problems with IT disrupted clinical workflow, wasted time, caused frustration, and led to use of a hybrid records system. Technical problems related to user interfaces, routine updates to software packages and drug databases, and migration of records from one package to another generated clinical errors.
McDonald et al. (2006)45		Case study	United States		CPOE	Inpatient	Computerized barcoding patient identification system was associated with a near miss. Human factors issues involved rule violation and integration with workflow, such as missing verbal confirmation of patient identification and entering wrong information into a system.
Meeks et al. (2014)26	2009–2013 (34)	Incident analysis (qualitative)	United States	100	EHR	Inpatient, outpatient, long-term care	A total of 70% involved 2 or more sociotechnical dimensions: (1) hardware and software, 76; (2) clinical content, 38; (3) human-computer interface, 29; (4) people, 20; (5) workflow and communication, 35; (6) internal organizational features, 6; (7) external rules and regulations, 2; (8) system measurement and monitoring, 1. Phase 1: unsafe technology or technology failures, 74. Phase 2: unsafe or inappropriate use of technology, 25. Phase 3: lack of monitoring of safety concerns, 1.
Menon et al. (2014)34	2012 (2)	Surveys (qualitative)	United States	369 responses	EHR		Responses from risk managers and health lawyers regarding the frequency and types of EHR-related serious safety events were investigated. In total, 53% of respondents reported at least 1 EHR-related serious safety event in the previous 5 years. Technical problems were related to software functionality such as screen display and data aggregation, configuration issues, and downtime. Use errors involved incomplete, missing, or wrong entries for orders. Consequences included failure to follow up abnormal test results and delays in accessing information.
Myers et al. (2011)22	1993–2010 (216)	Analyses of reports about equipment failure and hazards (qualitative)	United States	120	All health IT	Inpatient	A total of 120 technical problems were related to software functionality and system downtime. Consequences included delays in diagnosis or treatment and unnecessary or emergency procedures and/or treatment.
Nanji et al. (2011)40	2008 (1)	Descriptive (quantitative)	United States	466	ePS	Outpatient	In all, 10% of prescriptions in pharmacies (n = 3850) were associated with 466 medication errors. Error rates varied by computerized prescribing system, from 5.1% to 37.5%. The most common cause of error (61%) was omitted information such as duration, dose, or frequency. The remaining 39% of errors involved conflicting and incorrect information. One-third (n = 163) had potential for harm and 68 were serious, and none were life-threatening.
Odukoya et al. (2014)31		Interviews (qualitative)	United States	75, 107 interviews	ePS	Outpatient	CPOE in community pharmacies was associated with medication errors. The most common errors were wrong drug quantity, wrong dosing directions, wrong duration of therapy, and wrong dosage formulation. Potential consequences included additional work for pharmacy staff and frustration for patients due to delays. Technical problems included issues with system functionality (eg, dropdown menus) and system configuration issues.
Palchuk et al. (2010)39	2007 (3)	Descriptive (quantitative)	United States	470	EMR	Outpatient	Medication errors caused by inconsistent information between structured and free-text fields were identified in 20% of prescriptions (n = 2914); 84% of errors had the potential to lead to adverse events and 17% had the potential for severe harm, including death. Delivery of medications was delayed because pharmacy staff had to telephone prescribers to resolve conflicting information.
Samaranayake et al. (2012)19	2006–2010 (60)	Incident analysis (qualitative)	Hong Kong	243	CPOE, EHR, AutoDisp	Inpatient, outpatient	Most medication errors were related to prescribing and were linked to human factors issues. While most were detected before reaching patients, 11% did reach patients. Of the errors that reached patients, 6.1% caused minor injury and temporary morbidity requiring medical intervention.
Santell et al. (2009)18	2001–2005 (54)	Incident analysis (qualitative)	United States	90, 876	CPOE	Inpatient, outpatient, long-term care	Study focused on medication errors in hybrid record systems where paper records were used in combination with CPOE to process medication orders. Use errors such as partial/omitted information and rule violations were reported to be the leading cause of prescribing errors.
Schiff et al. (2015)21	2003–2010 (88)	Incident analysis (qualitative)	United States	10, 060	CPOE	Inpatient, outpatient	The majority of CPOE-related medication incidents were associated with use errors when entering orders. Human factors issues involved rule violations, lack of system and clinical knowledge, and communication issues. Technical problems were related to poor system functionality, interface with other systems. Errors were also attributed to hybrid record systems.
Singh et al. (2009)29	2007–2008 (4)	Descriptive (quantitative)	United States	633	CPOE	Inpatient, outpatient, pharmacy	Of 55 992 prescriptions, 532 (0.95%) were reported to contain inconsistent information between the structured template and the free-text field. Drug dosage was the most common element that was inconsistent. Human factors issues involved insufficient system knowledge and poor integration with clinical workflow. A total of 20% of errors had the potential for moderate to severe harm.
Slight et al. (2013)35	2010–2011 (8)	Interviews (qualitative)	England	34 interviews	CPOE	General practice	COPE used in general practice was associated with medication errors. Technical problems were related to poor user interfaces that led to selection errors. Unnecessary and inadequate alerts and slow system operation were also indicated as technical problems. Human factors issues were related to poor integration with clinical workflow and cognitive load due to multitasking and interruption.
Stewart et al. (2012)23	2005–2011 (82)	Incident analysis (qualitative)	Australia	21	All health IT	Inpatient	Most IT-related radiology incidents were associated with human-machine interactions occurring at data entry, transfer, and output. Incidents also involved use errors, such as uploading of wrong files and duplicated test orders.
Walsh et al. (2006)38	2002–2003 (9)	Observ (quantitative)	United States	20	CPOE	Inpatient	Twenty CPOE-related errors were identified among 104 medication errors in a pediatric setting. Of these, 7 were serious and 13 had little potential for harm. Use errors led to selection of wrong options from dropdown menus and wrong information entered using a keypad, generating duplicate orders. Eight medication errors were linked to wrong use of predefined order sets
Warm et al (2012)24	2009–2011 (29)	Incident analysis (qualitative)	UK	149	All health IT	Inpatient, outpatient, mental health, general practice	A total of 77% of incidents were linked to technical problems, eg, access issues, computer system down/too slow, display issues, and software malfunctions. In all, 10% related to human factors issues; 13% were not categorized.
Westbrook et al. (2013, 2012) 41,51	2008 (4) 2008–2010 (24)	Comparative (quantitative)	Australia	493	CPOE	Inpatient	System-related medication errors were frequent and the majority of errors manifested as timing errors. Only 11 errors were rated as potentially serious. While the system-related medication error rate was similar between 2 tertiary-care sites using different CPOE systems, there were significant differences in the underlying mechanisms for these errors. Selection, editing, and construction activities to write orders were the main mechanisms for error. Software functionality requiring new tasks from users also contributed to errors, eg, ordering a reminder or changing a default date. One system contained an incorrect order sentence.
Wetterneck et al. (2011)30	2006–2008 (5)	Comparative (quantitative)	United States	167	CPOE, CDS	Inpatient	CPOE implementation increased the duplicate medication order rate from 2.6% to 8.6%. Most post-implementation duplicate orders were for either the identical order or the same medication. Software issues were related to poor displays and issues with decision support. Human factors issues were related to lack of communication after shifts or during rounds. These issues contributed to duplicate orders for the same patient within minutes by a different physician.
Yu et al. (2013)33	2009–2011 (15)	Interviews (qualitative)	Australia	110 interviews	EHR	Long-term care facility	Eight types of unintended adverse consequences of EHR use were in residential aged care homes: inability/difficulty in data entry and information retrieval, end-user resistance to using the system, increased complexity of information management, end-user concerns about access, increased documentation burden, reduced communication, lack of space to place enough computers in the workplace, and increasing difficulties in delivering care services. Overall, technical problems were related to poor software functionality, which disrupted data entry/retrieval. Consequently, end users believed that EHRs increased documentation and disrupted care delivery.
Zhan et al. (2006)17	2003 (8)	Incident analysis (qualitative)	United States	7,029	CPOE	Inpatient, outpatient	Technical problems were related to faulty computer interface, miscommunication with other systems, and inadequate decision support. Human factors issues involved knowledge deficit, distractions, inexperience, and data entry errors.

Abbreviations: HIT: health information technology; observ: observational study; CPOE: computerized provider order entry; EMR: electronic medical record; EHR: electronic health record; ePS: electronic prescribing system; CDS: clinical decision support; AutoDisp: automated dispensation of medications; eMAR: electronic medication administration record; PIS: pharmacy information system; ICU: intensive care unit; ED: emergency department; PACS: picture archiving and communication system; GP: general practice.

Table 2.

Characteristics of included studies and reported effects of health IT problems by the stages in the information value chain

Characteristic/Effects of IT problems	Studies, n (%)
Study design and methods
Qualitative	27 (79)
Incident analysis	14 (41)
Case study	4 (12)
Surveys, interviews, participant observation	9 (26)
Quantitative, observational studies	7 (21)
Descriptive	4 (12)
Comparative	3 (9)
Setting
Inpatient	26 (76)
Outpatient	15 (44)
Long-term care	4 (12)
General practice	6 (18)
Country
United States	21 (62)
Australia	5 (15)
United Kingdom	3 (9)
Other	5 (15)
Type of IT system
All health IT	10 (29)
CPOE	17 (50)
Electronic health record	7 (21)
Ambulatory prescribing	2 (6)
Dispensing	2 (6)
IT problemsa	34 (100)
2.1 Hardware (device) down or slow12–15,20,24–26,32,33	10 (29)
2.2 Data capture/output peripheral device down or slow12–15,24	5 (15)
2.3 Network/server down or slow12–15,20,24,25,33	8 (24)
2.4 Software not available or not licenced12,13,15	3 (9)
2.5 Software not accessible12,13,17,22,24,27,28,33,34,37,42,46	12 (35)
2.6 Software issue13,18,25,26,32	6 (18)
2.6.1 Functionality (including user interface and task fit)12,14,15,17–20,22,23,26–28,30,31,33–44	26 (76)
2.6.2 System configuration (including decision support rules)12,14,15,17,18,20,22,24,26,27,30,31,34,36,38,40,42,44	18 (53)
2.6.3 Interface with devices12,14,15	4 (12)
2.6.4 Interface with other software systems or components12,15,17,20,21,24,26,28,32,37	10 (29)
2.6.5 Increased volume of transaction12,22	2 (6)
2.7 Data storage and backup12–14,24,27	6 (18)
2.8 Record migration12,15,22,23	4 (12)
2.9 Power failure12,25,32,33	4 (12)
2.10 Computer virus12,15,25,32	4 (12)
Contributing factorsa	24 (71)
3.1 Staffing/training12,13,15,17–19,21,23,26–29,32,34,35,37,40,43,44	19 (56)
3.2 Cognitive load15,17,27,28,35–37	7 (21)
3.2.1 Interruption12,15,35	3 (9)
3.2.2 Multitasking13,15,35	3 (9)
3.3 Failed to carry out duty15,18,19,21	4 (12)
3.3.1 Failed to log off12,13,27,32	4 (12)
3.4 Information governance12,17,23,26,27,32,34–36,43	10 (29)
3.5 Integration with clinical workflow12,18,26–30,34–37,42–45	15 (44)
Information receiveda	33 (97)
Use error12–15,17–41,44,45	31 (91)
1.1.1 Wrong12–15,17–24,27–31,34–37,39–41,44,45,47	26 (76)
1.1.2 Partial12–15,18,19,21,22,24,27,29,34,39–41	15 (44)
1.1.3 Missing12–15,18,20,21,24,27,34,40,41	12 (35)
1.1.4 Delayed15	1 (3)
Machine error12–15,17,18,20–24,27,28,30,34–38,42–44	22 (65)
1.2.1 Wrong12,15,18,20,22,24,27,28,30,34–38,42,43	16 (47)
1.2.2 Partial12,15,21,34	4 (12)
1.2.3 Missing12,14,15,17,20–22,24,34,38,44	11 (32)
1.2.4 Delayed12,15,27,35	4 (12)
Decision changed	27 (79)
Omission error12–15,18–21,24,27,30,34–36,38,40,41,45	18 (53)
Commission error12–15,17–22,24,27–31,34–41,44,45	26 (76)
Delay12,15,35	3 (9)
Care process altered	25 (74)
Clinical error12–15,17–22,24,25,29,31,32,34,38–41,44	15 (44)
Delay12,13,15,22,25,27,29,31,35,39,42–46	15 (44)
Outcome changed	21 (62)
Near miss12,13,15,17–19,32,38,41,45	10 (29)
Actual or potential patient harm12–15,17–20,22,24,25,29,32,38–41,44	18 (53)

^anumbers refer to categories from an earlier classification of safety problems associated with health IT¹²

User interaction

The first stage in the value chain was associated with multiple IT problems, typically technical and sociotechnical or context issues, and these also had the potential to affect multiple stages of the chain. Issues with accessing software were reported in 35% of the studies (eg, software was not available at a particular workstation, was not accessible, or did not have the correct settings). Other commonly reported IT problems were related to interfaces with other software (29%, n = 10), hardware malfunction (29%, n = 10), and network issues (24%, n = 8). Issues with software functionality appeared consistently across a majority of studies (Table 2). In 76% of studies, poor user interfaces and fragmented displays (eg, preventing a coherent view of all of a patient's medications) were associated with errors in selecting and entering information. Other software issues were related to system configuration, especially problems with default settings, which were reported in more than half of the studies (53%, n = 18). System configuration issues were also linked to software updates, eg, decision support errors following updates to a drug database.²⁶

Sociotechnical contextual variables that contributed to information errors were identified in 71% of studies (n = 24). The most commonly reported problem was staffing and training to use IT systems (56%, n = 19). Other contributing factors included integration with clinical workflow (44%) and information governance (29%), eg, procedures to authorize medications^27,36 and IT policy.^23,34

Information received

Information errors arising from the use of software were reported in most studies (91%, n = 31). For example, autopopulated fields in a prescribing system contained incorrect information such as drug dosing directions.³¹ Problems in data entry and retrieval were linked to wrong (76%, n = 26), partial (44%, n = 15), missing (35%, n = 12), and delayed (3%, n = 1) information. Five studies did not identify specific types of use errors, describing them as keypad or computer entry errors.^{25,26,32,33,38} In contrast, machine errors were reported in 65% of studies (n = 22). These were due to wrong (47%, n = 16), missing (32%, n = 11), partial (12%, n = 4), and delayed (9%, n = 3) display of information. For example, alerts about drug-drug and drug-allergy interactions failed to display (missing information) or were wrongly displayed (wrong information). As with data entry, some studies did not identify specific types of machine errors, describing them as display or data output errors (n = 2).^13,23

Decision changed

The effects of IT problems on errors and delays in clinical decision-making were reported in 76% of studies (n = 26). For example, errors in predefined order sentences led to clinical errors such as wrong dose and wrong route.⁴¹ Commission errors were the most commonly reported (76%, n = 26). These were linked to wrong data entry, selection from dropdown menus, and file uploads. In contrast, omission errors were reported in 50% of studies (n = 17) and delays in only 9% (n = 3). For example, users ignored alerts (n = 6) and failed to update information.^13,14 Many delays in clinical decision-making were linked to computer network issues.^12,15,35

Care process altered

The effects of IT problems on clinical errors and delays in care process could be identified in 44% of studies (n = 15). Examples of such errors include medication administration errors and failure to follow up test results. Delays in care process were linked to system access^24,25 and software functionality issues, including poor user interfaces and fragmented displays.²⁷ Issues with system configuration,⁴³ particularly software updates, were also reported to impact care delivery.⁴⁶ In 1 study, poor integration of an electronic health record with clinical workflow disrupted care delivery in a long-term care facility.³³ Other effects on care delivery included cancellation of patient appointments^24,25 as well as unnecessary or emergency clinical procedures and treatment.²² IT issues were reported to create more work for health professionals (21%, n = 7). For example, pharmacists needed to telephone clinicians to clarify IT-related errors and discrepancies in prescriptions.^27,29,39 IT problems also wasted time and caused frustration. In 1 study, primary care doctors reported spending 2 hours per week solving IT issues.¹⁵ Strategies for dealing with IT problems, including workarounds, were reported in many studies (21%, n = 7). For example, free-text fields were used to enter complex medication regimens when there were difficulties using CPOE systems.^27,41 Another commonly reported strategy was to revert to paper, creating a hybrid record system (15%, n = 5). For example, when orders for some medications (eg, those requiring a variable dose regimen) could not be entered electronically, they would be written on paper, creating an opportunity for information to be missed.^27,38

Outcome changed

Actual or potential patient harm was reported in 52% of studies (n = 18; Table 3). Patient deaths were reported in 7 studies; 6 of these were incident analyses.^{12,14,18,20,22,25} In the seventh study, CPOE implementation in a children's hospital and accompanying policy changes imposed by hospital management were associated with increased mortality, from 2.8% to 6.6%.⁴² After CPOE implementation, life-saving treatment for critically ill ICU patients was delayed because orders could not be entered unless patients were registered in the system. New workflows caused a breakdown in doctor-nurse communication, and changes to policies and procedures for dispensing and administering medications also delayed treatment. Only 2 studies provided details of patient deaths (Table 3). Of the 7 deaths for which details were available, 2 were linked to patient misidentification; 3 were associated with software use errors that resulted in failure to treat in 2 cases and a wrong procedure. The sixth involved a medication overdose due to poor software functionality. The seventh death was related to a delay in treatment following hospitalization because a pending test result from a previous hospitalization was not visible to the relevant clinicians.

Table 3.

Reported effects of incidents and errors on care delivery and patient outcomes

Study	Errors/ incidents (n)	Potential or actual harm (%)	Patient deaths (n)	Near miss (%)	Noticeable consequence but no patient harm (%)	No noticeable consequence (%)	Hazard (%)
Incident analyses and surveys
Cheung et al. (2013)20	668	58	Two deaths, no further details provided
Hanuscak et al. (2009)32	39	36	None reported	64
Lei et al. (2013)25	116	1	One death, no further details provided		94
Magrabi et al. (2010)13	68a	3	None reported	4	38	34	13
Magrabi et al. (2012)14 Magrabi et al. (2011)50	436	11	Four deaths linked to patient misidentification, failure to treat, wrong procedure due to software use errors, overdose due to poor software functionality		10	32	46
Magrabi et al. (2015)12	850	3	Three deaths linked to patient misidentification, failure to treat due to software use errors, delay in treatment following hospitalization because a pending test result from a previous hospitalization was not visible to relevant clinicians	4	24	1	68
Magrabi et al. (2015)15	90	7	None reported	27	43	7	17
Myer et al. (2011)22	120	80	Deaths noted but number not specified
Samaranyake et al. (2012)19	263a	11	None reported	89
Santell et al. (2009)18	90, 876	43	Three deaths, no further details provided	48			14
Warm et al. (2012)24	149	34	None reported
Zhan et al. (2006)17	7,029	5	None reported	32	63b
Observational studies
Han et al. (2005)42	548	7	Mortality increased from 2.8% to 6.7%, no further details provided
Nanji et al. (2011)40	466	35	N/A
Palchuk et al. (2010)39	470	84	N/A
Singh et al. (2009)29	558a	71	N/A
Walsh et al. (2006)38	20	50	N/A	50
Westbrook et al. (2012)51 and (2013)41	493	2	N/A

^aErrors analyzed for outcome were different from total errors reported.

^bIncludes Noticeable consequence but no patient harm(%), No noticeable consequence (%), and Hazard (%).

In 2 studies analyzing safety events reported to the US Food and Drug Administration and from across England’s National Health Service (NHS), human factors issues were proportionally higher in the events involving patient harm.^12,14 The potential of IT problems to lead to large-scale adverse events (ie, affecting multiple individuals) was reported in 2 studies.⁴⁸ One was a study of safety events across England's NHS, where 23% of events (n = 850) affected more than 10 individuals.¹² In the second study, 36% of system downtimes (n = 116) in China were estimated to affect more than 100 individuals.²⁵ Near-miss events were reported in 29% of studies (n = 10).

Quality of studies and risk of bias

All 34 studies utilized observational designs. Mean study duration was 41 months, with a range of 1–144 months. Study population was broadly characterized by interviewees and survey respondents. In the 5 studies using interviews, there were 72 participants on average (range: 32–110), and there were 210 respondents (range: 32–369) in the 4 studies using surveys. We found that the average number of errors across the 21 studies was 5,401 (range: 20–90,876).

The main risk of bias was that the majority of studies were not true observational studies where the frequency of events was representative of the population, but were studies of incident reports where frequency could not be correlated with true population incidence. Furthermore, incident reports are potentially biased to events that appear important to the reporter.⁴⁹ We assessed each study using the Cochrane Collaborations tool for assessing risk of bias. On average, data completeness was 74% across all stages in the information value chain. Machine errors (n = 12) and outcomes (n = 13) were frequently not reported. Only 7 studies (21%) provided information across all stages in the value chain, whereas 10 and 11 studies (29% and 32%) missed 1 and 2 stages, respectively. In 4 out of 5 studies (n = 28), user interactions, consequences, and medication error types were categorized.

DISCUSSION

Problems with IT are pervasive in health care. However, the evidence for IT-related disruptions to care delivery and risks to patient safety still comes largely from qualitative studies. Most of the evidence of patient harm comes from incident reports, with the exception of 1 comparative study where CPOE implementation was associated with an increased risk of mortality in a children's hospital.⁴² However, this finding was not replicated when the same system was implemented at different sites, and the disparity in outcomes was likely the result of differences in local implementation processes.⁵² More generally, only 2 other studies identified in this review were comparative, detecting an increase in duplicate medication orders³⁰ and new system–related prescribing errors following CPOE implementation.^41,51 While the types of IT problems have been well documented in the literature, further observational studies are required to measure their frequency and the magnitude of their impact on care delivery and patient outcomes.

Implications for measuring the effects of IT problems

The limited evidence on the magnitude of IT problems and their impact may indicate an underlying problem with measurement.⁴⁷ Existing classification frameworks tend to identify problems by their cause but not their effects,⁵³ whereas patient safety frameworks do allow us to assign broad categories of consequence, such as whether or not a patient harm is considered severe.

In this review, we have attempted to develop a model that links cause to effect within the clinical decision-making and care process, using the information value chain as a template. The chain assists in identifying which process is impacted by an IT problem, but also shows the many stages through which information errors can then propagate.

The value chain thus offers a simple yet potentially powerful way to pinpoint specific threats to patient safety and identify the effectiveness of existing system defenses and new measures required to deal with clinical errors associated with IT. Were it to be used as a standard template in future studies, it would assist in making comparative assessments between studies. For example, the value chain could be integrated into the Statement on Reporting of Evaluation Studies in Health Informatics guidelines.⁵⁴

The many reporting gaps identified in this review underline the need for a more structured approach to recording the causes and effects of IT problems in health care. It was not possible, for example, to quantify the propagation of information errors in the included studies, because these effects were not adequately described. Of the 34 studies we reviewed, only 44% (n = 15) reported effects at all stages of the chain. This is an inherent limitation, as each study has its own objective and may not consider all the different effects of IT problems along the value chain. For example, observational studies looking at medication errors may not look at patient outcomes.

Where the effects of IT problems were captured, data quality was poor. For example, IT problems, use errors, contributing factors, and clinical errors were not clearly differentiated.³⁸ In other cases, IT problems were combined with use errors and contributing factors.⁴¹ Use errors were also combined with medication errors,³⁹ and information errors were combined with decision-making errors.⁴⁰ Other issues were related to heterogeneity in measures, even for the reporting of medication errors, which are among the most commonly studied errors in patient safety. For example, some studies reported common clinical error types (eg, wrong dose, wrong timing, wrong route, etc.),⁴¹ while others examined the clinical impact of omitted, unclear, and conflicting information in prescriptions.⁴⁰

One way to improve data quality is to use existing schemas for describing and measuring variables along the value chain. For instance, our earlier classification is based on the natural categories of IT problems described in incidents from Australia, the US, and England.¹² It provides a validated and now widely used schema for characterizing IT problems, contributing factors, and information errors and can be used in conjunction with the value chain to enhance measurement. Uniform characterization of information errors and their impact on patient safety can also provide a common language to facilitate collaboration and sharing among organizations with disparate IT implementations so that the most significant risks to patient safety can be identified.

User interaction

The information value chain begins with clinical users interacting with information from IT systems before considering decisions and taking action. The different types of IT problems that could affect user interaction are similar to those described in our earlier classification for safety problems associated with health IT. No new categories were required to code the IT problems, information errors, and contributing factors identified in this review, further validating the classification. The frequency of IT problems and their scale and severity are areas for further investigation.

Information received

We found that use errors that interfered with the receipt of patient information were reported in a majority of studies and were commonly linked to poor training and lack of familiarity with the system. Poor user interfaces also contributed to use errors and were sometimes exacerbated by machine errors. For example, clinicians frustrated by multiple irrelevant alerts were reported to disregard all alerts.^26,36 Machine errors in displaying alerts and overalerting were reported in 7 studies, and 6 studies reported use errors involving wrongly overridden alerts. Information errors were poorly characterized. This may be inherent to the nature of incident reports, which made up the bulk of the studies reviewed. Incident reports only give a snapshot of safety events and are typically provided by clinicians, who gradually acquire the information required to make decisions and are accustomed to dealing with incomplete information. Therefore, information errors due to missing and partial information may not be available in incident data.

Decision changed

The impact of information errors on clinical decisions was not examined adequately in the studies reviewed here. Only 3 studies reported delays to decision-making due to IT problems. This figure may be underreported, given that software access problems were reported in many of the reviewed studies. For example, errors and delays in decision-making can occur when software is not available^12,13,15 or not accessible^{12,13,17,22,24,27,28,33,34,37,42,46} due to power failure^12,25,32,33 or computer viruses,^12,15,25,32 but these were not explicitly noted. While observational studies are needed to understand patterns, controlled laboratory experiments can be used to quantify the impact of information errors on decision-making.⁵⁵

Care process altered

We found that clinical errors were well characterized in areas such as medications.^{17–21,29,31,38–41} For example, prescribing error types were neatly identified in 1 study that examined system-related errors.^41,51 Taking the likelihood and impact of clinical errors into consideration, another study specifically sought to distinguish errors that were unique to IT as well as those that were made more likely with IT and more likely to cause harm with IT. Such approaches to understanding the nature of IT-related clinical errors will enable better targeting of strategies for prevention and mitigation. For example, errors that are no different from those found with paper records can be addressed by building upon existing patient safety initiatives. Patient identification is one such area where problems have existed with paper records due to gaps in local procedures, and these are likely to persist and propagate via electronic records. However, errors that are unique to IT and those that are more likely to occur or are more likely to cause harm may require new and innovative approaches.

IT problems hindering access to software delayed the initiation of clinical tasks, and software functionality issues delayed completion of clinical tasks. Such problems led to frustration for clinicians, wasted people’s time, and sometimes led to workarounds. Delays in care processes may have been likely in up to 21 studies where software was not available or accessible, or where power failures and computer viruses prevented access. The use of hybrid records may also be underreported. Only 5 studies (15%) reported using hybrid systems to work around IT issues. As part of contingency planning for planned and unplanned downtime, paper forms are generally used to document patient care and communicate with other departments.⁵⁶ However, use of such procedures was not reported.

Outcome changed

Study designs and issues with data quality did not allow quantitative analysis of outcomes. The majority of studies were incident analyses, which are useful to understand the types and consequences of safety problems with IT and examine typical patterns along the value chain. However, as incident reports do not represent a systematic sample, they cannot be used to quantify the impact of IT problems on care processes and outcomes.⁵⁷ Further observational studies are required to measure the frequency of the different types of IT problems and quantify their effects.⁵⁸ While the impact of problems at a large scale such as system downtime might be quantifiable, the effects of low-frequency problems affecting small numbers of patients might be harder to measure than in other domains in patient safety.

Limitations of this review

This review has several limitations. It was restricted to studies of IT systems for clinicians that were published in the biomedical literature. We did not include a range of other sources of information about IT problems in health care, such as medical record review, routine data collection, medicolegal investigations, complaints, etc.⁴⁹ It is thus possible that the IT problem types and effects are not exhaustive. Heterogeneity in study design and IT problem types prevented quantitative examination of effects on care delivery and patient outcomes.

CONCLUSION

This review confirms that problems with health IT can disrupt care delivery and harm patients. The research evidence is largely qualitative, and there remain many opportunities to systematically study and quantify IT risks alongside its benefits to patient safety. The information value chain can be applied prospectively to quantify the effects of IT problems on user interaction information received, decision-making, care processes, and outcomes.

FUNDING

This research is supported by Australian National Health and Medical Research Council Centre for Research Excellence in e-Health grant 1032664. The funding source did not play any role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the article for publication.

COMPETING INTERESTS

The authors have no competing interests to declare.

CONTRIBUTIONS

FM and EC conceptualized the study. MK and FM led the literature search and data analysis and drafted the paper. MK is responsible for the integrity of the work; she is the guarantor. All authors participated in writing and revising the paper. All aspects of the study (including design; collection, analysis, and interpretation of data; writing of the report; and decision to publish) were led by the authors.