Effects of health information technology on patient outcomes: a systematic review

Abstract

Objective To systematically review studies assessing the effects of health information technology (health IT) on patient safety outcomes.

Materials and Methods The authors employed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement methods. MEDLINE, Cumulative Index to Nursing Allied Health (CINAHL), EMBASE, and Cochrane Library databases, from 2001 to June 2012, were searched. Descriptive and comparative studies were included that involved use of health IT in a clinical setting and measured effects on patient safety outcomes.

Results Data on setting, subjects, information technology implemented, and type of patient safety outcomes were all abstracted. The quality of the studies was evaluated by 2 independent reviewers (scored from 0 to 10). A total of 69 studies met inclusion criteria. Quality scores ranged from 1 to 9. There were 25 (36%) studies that found benefit of health IT on direct patient safety outcomes for the primary outcome measured, 43 (62%) studies that either had non-significant or mixed findings, and 1 (1%) study for which health IT had a detrimental effect. Neither the quality of the studies nor the rate of randomized control trials performed changed over time. Most studies that demonstrated a positive benefit of health IT on direct patient safety outcomes were inpatient, single-center, and either cohort or observational trials studying clinical decision support or computerized provider order entry.

Discussion and Conclusion Many areas of health IT application remain understudied and the majority of studies have non-significant or mixed findings. Our study suggests that larger, higher quality studies need to be conducted, particularly in the long-term care and ambulatory care settings.

Effectively harnessing the potential of health information technology (health IT) to improve patient safety, reduce harm, and improve patient outcomes remains a unifying national goal among healthcare providers, patients, and regulators. For several decades, the use of computer systems has been considered a potential mechanism to support and improve clinical care. ¹ Through the Medicare and Medicaid Electronic Health Record (EHR) Incentive Program, known as the meaningful use program, the federal government is investing billions of dollars to promote the adoption of health IT in order to improve patient outcomes. ² Rates of health IT adoption in the inpatient and outpatient settings are increasing, and the range of available technology remains vast and varied. ³ An important barrier to health IT adoption has been the uncertain effect on patient outcomes, particularly given the costliness of implementation of computerized infrastructures. ⁴ In order to evaluate the current state of the literature, we conducted a systematic review to determine the effect of multiple health IT tools on patient safety outcomes.

While 31 systematic reviews have been conducted with a focus on health IT interventions and patient safety outcomes, this systematic review is different for several reasons. First, many of the systematic reviews focused upon one specific health IT, ^5–10 most commonly clinical decision support (CDS). ^11–26 Second, prior reviews often focused upon one area of clinical care such as outpatient, ^13,15,27,28 inpatient, ^6,16,29 intensive care, ³⁰ pediatrics, ^5,30 or geriatrics. ²³ Other papers targeted very specific outcomes, such as the effects of health IT as it relates to antibiotic medications, ²² anticoagulant therapy, ²⁰ lab testing, ⁷ or treatment of hypertension. ¹³ Finally, many prior reviews looked specifically at effects of health IT on one safety outcome—adverse drug events (ADEs). ^{5,6,19,24,26,29,31–33}

Prior studies generally included both non-randomized and randomized trials. ^{4–9,15–19,21–23,27–29,32–34} Eleven of the prior systematic reviews included only the highest level of evidence studies, randomized controlled trials (RCT). ^{10–14,20,24–26,31,35} Findings from these systematic reviews were mixed. Three of the previously mentioned 11 studies conducted a meta-analysis: 1 found improvement in patient safety outcomes, ²⁴ 1 stated insufficient studies to conclude, ²⁶ and the final paper was equivocal. ¹⁰

Therefore, this systematic review serves to provide a cumulative picture of the effects of multiple types of health IT on an array of direct patient safety outcomes in all clinical areas. This is an important time to be studying health IT as adoption rates continue to rise, policymakers continue to support and promote its use, and the determination of how and when to begin regulation of health IT remains under debate. To our knowledge, no prior systematic review has evaluated a comprehensive set of health IT tools while also exclusively focusing on determining the effects of those technologies on direct patient safety outcomes.

MATERIALS AND METHODS

Study Identification and Selection

Health IT was broadly defined as any automated or computerized system implemented to aid in the management of health information. We focused on the following health information technologies: computerized physician order entry (CPOE), e-prescribing, CDS, order entry alerts, EHR, health information exchange (HIE), patient portals, automated error detection software to detect medication errors (AED), electronic medication administration records (eMAR), medication administration barcodes, electronic medication reconciliation software (eMedRec), automated medication dispensing systems (AutoDisp), and electronic clinical pathways. Medication administration barcodes included barcode systems that dispense medication from an automated machine, as well as barcode systems that are used to ensure correct patient identification during the process of medication administration. Automated error detection systems referred to systems that look back to find the orders that may have led to an ADE or a pADE, in contrast to CPOE, which is designed to help aid the provider in correct prescribing at the point of care. We chose these tools through a combination of a priori knowledge of the literature, as well as health IT tools identified as part of the systematic review search process. Other patient-centered interventions such as health IT phone applications or home automated blood pressure cuff monitoring were not actively excluded; however, we did not identify any studies that assessed the impact of these technologies on direct patient outcomes. In cases in which authors did not identify the type of health IT employed using commonly known acronyms or terminology, reviewers used the description of the intervention to determine which type of health IT was being employed.

The authors also identified the clinicians under study. For cases in which the clinicians employing a particular health IT intervention were not identified, the authors reported “NR,” not reported. In cases where a clinician type was not applicable—for example, patient centered tools—those studies were denoted as N/A.

The patient outcomes chosen were identified from the studies included in the review, as well as from author knowledge of outcomes likely to be affected by health IT. After the analysis was completed, outcomes were then grouped on the basis of similar types of outcomes. In the articles for which more than one patient safety outcome was studied, reviewers included in the summary table only the primary outcome numerical effect size. However, for all outcomes, whether or not statistical significance was reached, the positive, negative, or non-significant effect on patient outcomes was considered and recorded ( Table 1 ).

Table 1:

Published Articles Studying Health IT Effects on Patient Outcomes.

Authors (year)	Study Period (months)	Study Design (Prosp or Retrosp)	Quality Score (0–10)	Country Study Site ( n )	Setting	Clinician Affected	No. Clinicians	Health IT Intervention	No. Patients	Patient Outcome a	Outcome	Effect Size b	P -value	Key Findings
Alvis et al. (1985). 36	NR	RCT (Prosp)	6	US Hosp (1)	OR	MDs	–	Smart Pump	30	Hemodynamic Stability (hypo- and hypertension)	P	ARR 14%	<.05	Patients who had computer-assisted, smartpump-mediated fentanyl infusion had greater hemodynamic stability. Fewer adjuvant drugs were used with smartpump technology than with manual bolus of fentanyl approach.
Barenfanger, Short, and Groesch (2001). 37	1998–1999 (5)	Cohort (Prosp)	3	US Hosp (1)	All wards	MDs	–	CDS	378	Mortality a LOS	P	ARR 1.4%	–	Decreased mortality and LOS were observed when adjusted for severity of cases when using CDS surrounding antibiotic usage. Costs were also decreased.
Buising et al. (2008). 38	2000–2007 (84)	Time Series (Prosp)	5	Australia Hosp (1)	All wards	MDs	–	CDS	189	GN bacteremia a In-hosp mortality LOS	M	NS	–	30-day mortality, rates of GN bacteremia, and LOS remained unchanged when CDS was employed to encourage appropriate antibiotic usage.
Chapuis et al. (2010). 39		Cohort (Prosp)	4	France Hosp (1)	Medical ICU	RNs	–	AutoDisp	115	ADE	M	NS	–	While automated dispensing system caused fewer medication errors as compared to baseline and concurrent control units, it did not reduce errors causing patient harm. Subjectively, nurses reported improved working conditions with automated dispensing system.
Chertow et al. (2001). 40	1997–1998 (8)	Cohort (Prosp)	8	US Hosp (1)	All adult wards	MDs	–	CDS CPOE	17 828 admissions	Renal Impairment a LOS	M	NS	–	There was no significant decrease in their primary outcome of the number of patients who experienced a decline in function during hospitalization. However, there was a significant reduction in length of stay during intervention periods with physician alerts to adjust medications based on renal function.
Cho et al. (2012). 41	2009–2010 (11)	Observ (Prosp)	2	Korea Hosp (1)	All adult wards	MDs	–	CDS	285	Contrast-induced AKI	P	ARR 7%	–	Rates of contrast-induced AKI dropped after implementation of a computer alert program reminding providers to order prophylaxis for patients with an elevated GFR undergoing a CT.
Cook et al. (2011). 42	2005–2009 (60)	Observ (Retro)	3	US Hosp (1)	All adult wards	MDs	–	CDS EHR CPOE	3612 charts	Nosocomial C. diff a Nosocomial MRSA	M	NS	–	Rates of nosocomial C. difficile infections did not change, but rates of MRSA infections decreased significantly after implementation of a EHR with CDS.
Colpaert et al. (2006). 43	2004 (1.25)	RCT (Prosp)	6	Belgium Hosp (1)	Surgical ICU	IntensivistsResidents	5 3 8 Total	CPOE CDS	90	Non-intercept potential ADEs a Rate of Non-intercepted potential ADEs Total ADEs Rate of ADEs	P	ARR 1.4 rates of Non-intercepted ADEs	<.001	The potential ADEs and actual ADEs were reduced in both absolute number and rate in units which CPOE with CDS was used as opposed to paper-based units. CPOE with CDS was found to have the most impact on medication prescribing errors that had the least potential negative patient impact.
Del Beccaro et al. (2006). 44	2002–2004 (26)	Observ (Prosp)	6	US Hosp (1)	Pediatric ICU	MDs	–	CPOE	2533	Mortality	M	NS	–	No significant change in mortality post CPOE implementation in a PICU was found when morality was tracked for 13 months pre- and post- implementation.
Dowding, Turley, and Garrido. (2012). 45	2003–2009 (84)	Time series (Prosp)	4	US Hosp (29)	All wards	RNs	–	EHReMARCPOECDS	NR	Hospital-acquired pressure ulcers a Fall injuries	M	NS	–	EHR implementation with risk assessment tools was associated with decreased rates in HAPU, but not in fall rates, but finding was not significant when controlled for over time.
Dykes et al. (2010). 46	2009 (6)	RCT (Prosp)	8	US Hosp (4)	Medicine wards	RNs	–	CDS	10 264	Fall Rates	P	ARR 1.03 falls per 1000 patient days	–	Use of CDS fall prevention tool was associated with decreased fall rate in intervention units. Patients over 65 years old had greatest decrease in fall rates. There was no significant change in falls resulting in patient injury.
Etchells et al. (2011). 47	2008 (6)	Cohort (Prosp)	7	Canada Hosp (2)	Medicine wards	MDs	–	CDS AED	271	Rates AEs	M	NS	–	Real-time alerting system with simultaneous decision support, but had no significant impact on rates of adverse events.
Evans et al. (1998). 48	1992–1995 (36)	Observ (Prosp)	6	US Hosp (1)	ICU	MDs	–	CDS	1681	ADEs a LOS	P	70% reduction	.018	Significant reduction in the rate of ADE from antibiotics was reduced by an anti-infective CDS tool. LOS was also decreased.
Evans et al. (1995). 49	1994–1995 (19)	Observ (Retro)	6	US Hosp (1)	Adult ICU	MDs	–	CDS	962	ADEs a	M	NS	–	Using a CDS system for antibiotic selection did not decrease rates of ADEs.
Evans et al. (1993). 50	1989–1992 (48)	Observ (Retro)	7	US Hosp (1)	All adult wards	MDs	–	AED CDS	92 649	ADEs	P	ARR 13.6%	<.001	Computerized surveillance of ADEs reduced the number of severe ADEs. Additionally, their surveillance system was used to create computer alerts to pharmacists when physicians prescribed medications to patients with previously known drug allergies, which also significantly reduced the rate of ADEs.
Fitzmaurice et al. (1996). 51	1993–1994 (12)	RCT (Prosp)	5	US Amb (2)	Primary Care	MDs	–	CDS	49	Mortality a Thrombotic events Hemorrhagic events	M	NS	–	There was no difference in adverse event rates for patients who were randomized to CDS dosing of their warfarin. INR was better controlled in intervention group.
Frances et al. (2001). 52	1997 (4)	RCT (Prosp)	4	US Amb (2)	Primary Care	MDs	63	CDS	730	Mortality a MI	M	NS	–	Rates of MI and mortality were not significantly different for patients whose physicians were randomized to have CDS reminding them to order appropriate post-MI medical management.
Fiumara et al. (2010). 53	2006–2008 (23)	Cohort (Prosp)	8	US Hosp (1)	Medical and Surgical wards	MDs	425	CDS	880	90-day incidence of VTE	M	NS	–	CDS to encourage DVT prophylaxis did not decrease rates of VTE at 90-days.
Fleming et al. (2009). 54	2006–2008 (30)	Cohort (Prosp)	2	US Hosp (8)	Medical wards	MDs	–	CDS Alg	4454	Inpatient Mortality a 30-day mortality	M	2.9% ARR	<.01	Unadjusted results showed reduction in both in-hospital mortality and 30-day mortality when order set was used. Results adjusted for covariates were not found to be significant with use of CDS. Adjusted results were of borderline significance.
Gandhi et al. (2005). 55	1999–2000	Cohort (Prosp)	6	US Amb (4)	Internal Medicine	MDs	34	CPOE	661	ADEs	M	NS	–	Rates of ADE were the same in the CPOE vs. handwritten medication orders in the ambulatory care setting. Non-significant trend towards increase in ADE was found.
Glassman et al. (2007). 56	2001–2002 (8)	Cohort (Prosp)	5	US Amb (NR)	Internal Medicine	MDs	–	AED CPOE	913	ADEs	M	NS	–	Rates of ADEs were the same in the retrospective error detection system for a CPOE system. The AED focused primarily on drug-drug interactions and drug-disease interactions.
Graumlich et al. (2009). 57	2004–2007 (39)	RCT (Prosp)	8	US Hosp (1)	Medicine wards	MDs	69	eMedRecCPOE	631	Readmission rate a ED visit rate ADEs post-discharge	M	NS	–	Discharge software did not affect readmission rates, ED visit rates or adverse events post-discharge.
Gurwitz et al. (2008). 58	(12)	RCT (Prosp)	5	US and Canada LTC (2)	All wards	MDs PA NPs	37	CPOE CDS	1118	ADE rate a Preventable ADE rate	M	NS	–	CPOE with CDS did not reduce adverse drug event rate or preventable adverse drug event rate.
Han et al. (2005). 59	2001–2003 (18)	Observ (Retro)	6	US Hosp (NR)	Pediatric ICU	MDs	–	CPOE	1942	Mortality	Neg	OR 3.71	–	There was an unexpected increase in mortality coincident with CPOE implementation from 13 months before implementation and 5 months after CPOE implementation.
Holdsworth et al. (2007). 60	2000–2001, 2004 (9+7)	Cohort (Prosp)	5	US Hosp (1)	Pediatric ICU	NR	–	CPOE CDS	Baseline: 1210 post-CPOE: 1197	Rates of ADE a Total ADEs Rates of potential and preventable ADEs	P	OR 0.76	–	There was significant reduction in the total ADEs, preventable ADEs, and potential ADEs after implementation of CPOE system. Sub-group analyses found that there were significant reductions in adverse events associated with certain antibiotic drug classes. LOS remained nearly identical between the ADE and potential ADE groups both before and after CPOE implementation.
Jani, Barber, and Wong. (2010). 61	2005–2006 (13)	Cohort (Prosp)	7	UK Hosp (1)	Pediatric wards	MDs	–	CPOE	1590	Med dosing errors varying severity	P	ARR 1%	<.001	Electronic prescribing can reduce dosing errors without CDS.
Jha et al. (2008). 62	2004–2005 (24)	Cohort (Prosp)	4	US Hosp (1,672)	Medicine wards	N/A	–	CPOE	NR	Adjusted 30-day Mortality Rates for: (1) AMI a (2) CHF (3) PNA	M	ARR 1.5%	<.002	Hospitals that had implemented CPOE and participated in reporting were found to have lower rates of adjusted 30-day mortality in both AMI and PNA patients. There was no difference in mortality in CHF patients.
Jimenez-Munoz et al. (2011). 63	2006–2007 (4)	Cohort (Prosp)	7	Spain Hosp (1)	Medicine wards	MDs RNs	–	AutoDisp CPOE	NR	AEs	P	ARR 2.73%	<.003	CPOE and automated dispensing systems reduced medication errors affecting patient in two phases drug administering phases: transcription and administration when compared with the paper-based group. Error prevalence rates requiring monitoring or not, were not affected in the prescription phase.
Keene et al. (2007). 64	1995–1997 (39)	Observ (Prosp)	4	US Hosp (1)	Pediatric and Neonatal ICUs	MDs	–	CPOE	1291	Mortality	M	NS	–	Rates of mortality did not change with CPOE implementation.
King et al. (2003). 65	1993–1996 1997–1999 (72)	Cohort (Retro)	3	Canada Hosp (1)	Pediatric: Medical and Surgical wards	MDs Residents	–	CPOE	36 103	AEs	M	NS	–	There was no effect of rate ratios of ADEs in the pre- and post-CPOE implementation periods. There was a significantly decreased rate of potential ADEs in the control wards (with handwritten orders). By contrast, medication error rates were also significantly reduced in the CPOE wards.
Kucher et al. (2005). 66	2000–2004 (41)	RCT (Prosp)	4	US Hosp (1)	Medical and Surgical wards	MDs	120	CDS	2506	VTE at 90 days a Mortality at 30 and 90 days	P	3.3% ARR	<.001	For patients for whom thromboembolic prophylaxis had not been ordered, a computer generated alert to physicians reduced VTE rates at 90 days with no difference in mortality between groups.
Kuperman et al. (1999). 67	1994–1995 (4)	RCT (Prosp)	8	US Hosp (1)	Adult Medical and Surgical wards	MDs	–	CDS	NR	Mortality a Arrest Transfer to ICU MI Delirium Stroke Renal Insufficiency AKI Dialysis Return to OR All outcomes	M	NS	–	There was no difference in adverse event rates for patients who were randomized to having their physicians receive alerts regarding concerning laboratory values in their medical records. The intervention group had shorter median time intervals before an appropriate treatment was ordered.
Lecumberri et al. (2008). 68	2005–2007 (18)	Observ (Prosp)	6	Spain Hosp (1)	Medical and Surgical Wards	MDs	–	CDS	19 338	VTE	M	OR 0.53	–	Rates of VTE were not significantly reduced by alerts to physicians overall in the post-intervention period, but in sub-group analysis of surgical patients, a significant reduction in VTE events was identified, which was stable over time.
Lesourd et al. (2002). 69	2001 (NR)	RCT (both Retro and Prosp)	3	France Hosp (1)	REI	MDs	–	CDS	53 retrosp. 164 prosp.	Pregnancy	P	NS	–	CDS was as effective as clinicians in using ovarian stimulation with FSH in resulting in pregnancy.
Linares et al. (2011). 70		Cohort (Prosp)	3	US Hosp (1)	Medical wards	MDs	–	CDS	251	Antimicrobial-AE a C.diff from treatment	P	NNT 10	–	Asymptomatic bactiuria and culture-negative pyuria had decreased complications when a computerized alert reminded providers that those U/A and culture results did not require treatment.
MacIvor et al. (2009). 71	2005–2007 (31)	Observ (Retro)	7	US Hosp (16)	Medical wards	NR	N/A	HIE	16	Mis-transfusions	P	38% increase	–	Centralized patient database detected 38% more ABO typing errors and prevented 6 mis-transfusions
Madaras-Kelly et al. (2006). 72	2001–2004 (36)	Time series (Prosp)	1	US Hosp (1)	Medical and Surgical ICU	MDs	–	CDS	NR	MRSA Infection rates	P	0.0074% ARR	.02	Rate of nosocomial MRSA infections decreased using a CDS intervention to decrease flouroquinolone use. Rate of nosocomial gram-negative organisms significantly increased by 22.66%. Rate of trimethoprim-sulfamethoxazole and piperacillin-tazobactam increased in usage.
Maynard et al. (2010). 73	2005–2007 (36)	Observ (both Retro and Prosp)	5	US Hosp (1)	All adult wards except psych and ob/gyn wards	MDs	–	CPOE CDS	2924	Hospital-acquired VTE a health IT PPX-related bleeding	P	39% RRR	–	There was a reduction of the rate of HA VTE after introduction of CPOE and CDS. Neither health IT nor prophylaxis related bleeding was increased by this intervention.
McCowan e al. (2001). 74	(6)	RCT (Prosp)	4	UK Amb (17)	Primary Care	MDs	–	CDS	477	Acute asthma exacerbations a Hospitalization ED visits	M	OR 0.43	–	A significantly lower number of patients experienced asthma exacerbations if their physicians used CDS. Hospitalizations and ED visits had no effect.
McMullin et al. (2006). 75	1999 (3) 2001–2002 (12) 2002–2003 (3)	Cohort (Prosp)	3	Canada Hosp (1)	ICU	MDs	–	CDS	Phase 1: 68 Phase 2: 261 Phase 3: 101	VTE rates	M	NS	–	DVT and pulmonary embolism rates were similar for all phases despite increased compliance with prophylaxis guidelines in phases 2 and 3. Phase 2 was characterized by behavioral approaches plus computerized alerts, while Phase 3 consisted of alerts alone.
McMullin et al. (1999). 76	1994–1997 (48)	Observ (Retro)	3	US Hosp (1)	All wards	MDs	–	CDS	286	ADE	M	NS	–	Using medication alerts did not significantly reduce the rates of adverse events as related to drug-interactions. Rates of dangerous drug combinations and length of time for which those combinations were applied was reduced.
Menachemi et al. (2007). 77	NR	Observ (Retro)	5	US Hosp (98)	All wards	N/A	–	No. of IT Applications	NR	Eight patient safety indicators (PSIs)	M	−1.82 RRR in rates of mortality	.024	The greater the number of clinical IT applications adopted by a given hospital, the lower the adverse event rates in three of the eight PSIs: death in low-mortality DRGs, decubitus ulcers, and post-operative sepsis. The other five indicators did not reach statistical significance.
Milani et al (2011). 78	2009–2010 (24)	Cohort (Prosp)	5	US Hosp (1)	Medical wards	MDs	35	CDS CPOE	47 written orders 33 CPOE	In-hospital bleeding a LOS 90-day mortality	M	ARR 52%	.002	CPOE with decision support reduced hospital bleeding among patients with CKD admitted with ACS. LOS and 90-day mortality were not affected.
Morriss et al. (2011). 79	2005–2006 (10)	Observ (Prosp)	3	US Hosp (1)	Neonatal ICU	RNs	–	Barcode	618	Preventable ADE associated with opioid administration	P	0.48	.045	Barcode medication administration system was found to significantly reduce the risk of preventable ADEs associated with opioid administration.
Morriss et al. (2009). 80	NR (12.5)	Cohort (Prosp)	9	US Hosp (1)	Neonatal ICU	RNs and RTs	–	Barcode eMAR	958	Preventable ADEs	P	ARR 3%	.04	The barcode medication administration system was found to significantly decrease the risk of preventable ADEs.
Novis et al. (2010). 81	2007–2008 (12)	Observ (Prosp)	3	US Hosp (1)	Surgical wards	MDs	–	CDS	800	Post op DVT at 30 days a at 60 days and at 90 days	M	NS	–	After implementation of CDS for VTE prophylaxis postoperatively, there was a trend toward decreased rates of post-operative VTEs, which did not reach significance, due to not being powered to do so. Rates of DVT prophylaxis ordering increased.
Oliven et al. (2002). 82	(6)	Cohort (Prosp)	6	Israel Hosp (1)	Two internal medicine wards	MDs	NR	CPOE	1350	PEs prevented	M	ARR 31%	<.001	Although no the primary outcome, it was found that CPOE significantly reduced the rate of PEs.
Overhage et al. (2002). 83	1995–1996 (12)	RCT (Prosp)	7	US Hosp (2)	ED	MDs	72	HIE	32 468	Hospital admission a Repeat ED visits	M	NS	–	There was no difference in admission rates or repeat ED visits for patients whose ED physicians had access to electronic medical records from another institution. Few of the physicians actually made use of the online system to check OSH records.
Parente and McCullough. (2009). 84	1999–2002 (48)	Observ (Retro)	5	US Hosp (NR)	Surgical wards	MDs RNs	–	EHR CDS eMAR	National sample Medicare claims data	Infection due to medical care a Post-op hemorrhage Post-op VTE	M	NR	–	Rates of hospital-associated infections significantly decreased with EHR use, but none of the other outcomes were reduced by either EHR or the other health IT tools studied.
Paul et al. (2006). 85	2002–2004 (14)	RCT (Prosp)	5	Israel, Germany, and Italy Hosp (3)	All wards	MDs	199 cohort, NR RCT	CDS	350 in cohort study/2326 in RCT	LOS a 30-day mortality	M	NS	–	At two of three sites, LOS was significantly reduced by using CDS for antibiotic selection. Overall, there was no significant reduction. 30-day mortality in the intention to treat analysis was not significantly different either at any of the individual sites nor overall.
Peterson et al. (2005). 86	2001–2002 (6)	Cohort (Prosp)	3	US Hosp (1)	Medicine and ICU wards	MDs	–	CDS CPOE	3718	Hospital Fall rate a LOSDays of AMS	P	0.0036 ARR falls per 100 pt days	.001	There was a significant reduction of patient in-hospital fall rates after implementation of CDS. No effect was found on hospital length of stay or days of altered mental status.
Piontek et al. (2010). 87	2001	Observ (Retro)	5	US Hosp (7)	All adult wards	Pharmacists	–	AED	NR 230 000 admissions	Severity-adjusted Mortality rates a LOS Readmit Rates	P	decrease (amount NR)	<.001	The primary outcome of severity-adjusted mortality rates was significantly lower in hospitals with AED as compared with concurrent control group and pre-intervention group. However, there was no significant difference in LOS or readmission rates.
Poon et al. (2010). 88	2005 (9)	Observ (Prosp)	9	US Hosp (1)	All adult wards including ICU	RNs	–	Barcode eMAR	1726	Potential ADEs	M	ARR 1.5%	<.001	The rate of potential adverse drug events decreased significantly with use of barcode and eMAR. The rate of potential adverse events associated with timing errors did not change significantly.
Potts et al. (2004). 89	2001 (2) 2002 (2)	Observ (Prosp)	9	US Hosp (1)	Pediatric wards	MDs	–	CPOE	514	Potential ADEs	P	ARR 0.9 potential ADEs per 100 order	<.001	Rates of potential ADEs were significantly reduced after CPOE implementation.
Rollman et al. (2002). 90	1997–1998 (20)	RCT (Prosp)	1	US Amb (NR)	Primary Care	MDs	17	CDS EHR	226	Remission of depression at 3 a and 6 months	M	NS	–	There was no significant reduction in rates of recovery from depression when CDS was added to an EHR to actively remind physicians to manage patients' MDD.
Ross et al. (2004). 91	2001–2002 (13)	RCT (Prosp)	2	US Amb (1)	Cardiac	RNs MDs	–	Patient Portal	107	Mortality a Hospitalizations ED visits	M	NS	–	Increased medication adherence approached but did not reach significance in the intervention group using a patient portal. The number of patients who visited the ED was not different, but the number of visits was higher in the intervention group. Mortality and hospitalizations were also not significantly different.
Rind et al. (1994). 92	1990–1991 (18)	Time Series (Prosp)	2	US Hosp (1)	All adult wards	MDs	–	CDS	562	Renal impairment	P	RR 0.45	–	When computer-based alerts were activated during the intervention periods the rates of serious renal impairment was diminished.
Rothschildet al. (2005). 93	2002 (11)	Time series (Prosp)	3	US Hosp (1)	Cardiac Surgical ICU and step-down units	RNs	–	Smart Pump	735	Serious ADEs a Non-intercepted potential ADEs	M	NS	–	There was no measurable impact of smart pumps on the serious medication error rate and non-intercepted potential adverse drug events. Authors postulated that lack of compliance played a factor in the lack of effectiveness observed.
Schnipper et al. (2009). 94	2005–2006 (12)	Observ (Prosp)	6	US Hosp (1)	1 Medical ward	Pas	2	CDS CPOE	169	Percentage of patient days with Hypoglycemia a LOS	M	NS	–	Patient days of hypoglycemia did not change by adding CPOE to the admission order set for diabetic patients. Adjusted LOS did decrease by 25%.
Shulman et al. (2005). 95	2001–2002 (15)	Observ (Prosp)	9	UK Hosp (1)	ICU	MDs	–	CPOE	387	ADEs	M	NS	.51	Rate of major or moderate errors affecting patients were unchanged in the CPOE group vs the handwritten prescription group. There was a reduction of major/moderate patient outcomes when non-intercepted and intercepted errors were combined (.01).
Small et al. (2008). 96	2005 (5)	Cohort (Prosp)	6	UK Hosp (1)	Oncology wards	MDs	3	CPOE	NR	ADEs	P	ARR 8.6%	<.0001	CPOE reduced adverse drug events. Error type distribution differed significantly among CPOE orders vs manual orders. CPOE was associated with fewer dose calculation error rates. CPOE decreased the rate of minor errors, with higher proportion of significant and life-threatening errors relative to total errors.
Smith et al. (2010). 97	2006–2009 (31)	RCT (Prosp)	2	UK Amb (29)	Primary Care	MDs	–	CDS	911	Exacerbations a Hospitalizations	M	NS	–	There were no significant differences in exacerbations with use of CDS. Disaggregation of the composite outcome demonstrated that intervention of alerts was found to reduce the odds of patients requiring hospitalization. Rates of ED visits, after hours contacts with physicians was not significantly different. Rates of prednisolone use were increased by intervention.
Tierney et al. (2005). 98	1994–1996 (36)	RCT (Prosp)	6	US Amb (4)	Primary Care	MDs and Pharmacists	274 MDs 20 pharmacists	CDS	706	ED visits a Hospitalizations	M	NS	–	There was no significant difference in ED visits or hospitalizations rates between COPD patients of physicians who had CDS and those who did not.
Upperman et al. (2005). 99	2002 (9)	Cohort (both Retro and Prosp)	4	US Hosp (1)	All pediatric inpatients	MDs	–	CPOE	NR	ADEs	M	NS	–	Although in aggregate, rates ADEs were not significantly different after CPOE implementation, harmful ADEs were significantly reduced with CPOE usage with a NNT of 1 ADE per 64 patient-days.
Van Doormaal et al. (2009). 100	2005–2008 (37)	Cohort (Prosp)	5	The Netherlands Hosp (2)	Two hospital wards per hospital	MDs	NR	CPOECDS	NR	pADEs a AEs	M	NS	–	CPOE in combination with CDS was not statistically associated with a reduction in preventable ADEs.
Walsh et al. (2008). 101	2001–2002 (16)	Time Series (Prosp)	5	US Hosp (1)	All pediatric wards	MDs	–	CPOE	NR	Serious Medication Errors	M	NS	–	Overall, rates of non-intercepted serious medication errors were not significantly less. In the NICU and PICU, but not in the general pediatric wards, serious medication errors were significantly less after CPOE implementation. Time series analysis demonstrated that rates of medication errors varied significantly with time of year, irrespective of CPOE implementation, in which months earlier in the academic year yielded high error rates.
Weingart et al. (2009). 102	2006 (6)	Observ (Prosp)	4	US Amb (NR)	Adult Primary Care, Pediatric, Psychiatry, and other specialties	MDs and physician-extenders	2321	CDS	60 352	pADEs a disabilityhospitalizations ED visits office visits	P	331 alerts needed to prevent 1 ADE	–	Alerts were responsible for preventing ADEs as well as hospitalizations, ED visits, and office visits.
Yu et al. (2009). 103	2005–2006 (12)	Case-control (Retro)	3	US Hosp (122)	All pediatric wards	N/A	–	CPOE	1 151 932	ADEs	P	OR 1.42	–	Hospitals without CPOE had 42 percent higher rates of ADEs.
Zanetti et al. (2003). 104	2000 (4)	RCT (Prosp)	1	US Hosp (1)	OR	MDs	–	CDS	449 cases	Rate of surgical site infections	M	ARR 2%	.4	Surgical site-infection rates were reduced compared with pre-study period, but not significantly reduced compared to concurrent controls when CDS was used to alert physicians to give intraoperative antibiotics.

^a Primary outcomes noted with asterisks. ^b All effect sizes are reported for the primary outcome listed.

Abbreviations: Prosp = prospective; Retrosp or Retro = retrospective; Health IT = Health information technology; Conf Int = confidence interval; Observ = observational study; RCT = randomized control trial; Hosp = hospital setting; Amb = ambulatory care setting; NR = not reported; NS = not significant primary outcome; LTC = long term care facility; ICU = intensive care unit; REI = reproductive endocrinology and infertility; ED = emergency department; OR = operating room or odds ratio; MDs = physicians; RNs = nurses; PA = physicians’ assistants; NPs = nurse practitioners; N/A = not applicable; CDS = clinical decision support; CPOE = computerized provider order entry; EHR = electronic health record; HIE = Health information exchange; AutoDisp = automated dispensation of medication; eMAR = electronic medication administration record; AED = automated error detection system; eMedRec = electronic medication reconciliation; Alg = electronic clinical pathway; GN = gram negative; LOS = length of stay; C. diff =  Clostridium difficile ; MRSA = Methicillin-resistant Staphylococcus aureus ; ADE = adverse drug event; AE = adverse event; pADE = preventable adverse drug event; AKI = acute kidney injury; AMI or MI = acute myocardial infarction; CHF = congestive heart failure; PNA = pneumonia; VTE = venous thromboembolism; PPX = prophylaxis; DVT = deep venous thrombosis; PE = pulmonary embolism; post-op = post-operatively; AMS = altered mental status; M = not significant or mixed results study; P = positive study; Neg = negative study (health IT was found to be harmful); COPD = chronic obstruction pulmonary disease; ARR = absolute risk reduction; NNT = number needed to treat.

We performed searches in bibliographic databases, Ovid Medline, Ovid EMBASE, the Cumulative Index to Nursing Allied Health (CINAHL) via Ebscohost, and Cochrane Library from January 2001 to June 2012. Conference proceedings were reviewed as well as bibliographies of selected articles. Citations of all identified prior systematic reviews were also reviewed. The search strategy included combinations of keywords and controlled vocabulary. A validated filter to represent patient safety was applied. ¹⁰⁵ Appendix A illustrates the detailed search strategy for the four databases .

All citations, index terms, and abstracts (if available) were reviewed and rated as “potentially relevant” or “not relevant.” In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting of systematic reviews, one reviewer reviewed the entire set first of titles, followed by the abstracts. Articles that were potentially relevant were included in the set reviewed by 2 independent reviewers (see Figure 1 ). ¹⁰⁶ Articles were reviewed independently, and studies were included in the review if 1) the study participants were health professionals in clinical practice or postgraduate training, 2) the intervention was health IT studied in a clinical setting, and 3) the outcomes (even if secondary and not primary) that were assessed included at least one direct patient safety outcome (including any aspect of patient well-being, with process measures considered insufficient). Only English-language studies were included. All disagreements were resolved by consensus.

Figure 1:

Study identification and selection.

Study Evaluation

Two authors independently assessed all selected studies for methodological quality. A previously described 10-point Methodological Quality Assessment was adapted to the purposes of this study. ^9,11,17 This methodological rating scale assesses for 5 potential sources of bias, each scored either 0, 1, or 2, including (A) the method of allocation to study groups (random vs selected concurrent controls vs non-concurrent controls), (B) the unit of allocation (ward or clinic vs physician vs patient), (C) baseline differences between groups which could potentially be linked to the study outcome (no baseline differences and/or appropriate statistical adjustments made for differences vs baseline differences apparent without statistical adjustment vs unable to assess), (D) the type of safety outcome measure (objective outcome or subjective outcome with blinded assessment vs objective outcome with no blinding vs subjective outcome without blinding of assessors), and (E) completeness of follow-up (>90% vs 80%-90% vs <80% and/or unable to assess). ¹¹ As such, a score of 10 represents studies whose design had the lowest amount of bias ( Table 2 ). Disagreements were resolved by discussion to reach consensus. Reviewer agreement and inter-rater reliability was analyzed by the kappa statistical method. Since one reviewer reviewed all of the articles, and multiple reviewers were paired with the principal reviewer, a quadratic-weighted kappa was chosen. ¹⁰⁷

Table 2:

Study Quality Rating Scale

Potential Source of Bias	Score
	2	1	0
Allocation Bias	Randomized	Quasi-randomized	Concurrent controls
Unit of Allocation Bias	Cluster-analysis (i.e.,: practice or ward)	Physician-based analysis	Patient-level analysis
Baseline Group Characteristics	No baseline differences or appropriate statistical adjustments for differences	Baseline differences present with no statistical adjustments	Baseline differences not reported
Objectivity of Outcome	Objective outcomes with blinded assessment	Objective outcomes without blinding	Subjective outcomes with no blinding and poorly defined
Completeness of Follow-up	>90%	80–90%	<80% or not described

Adopting the methodology employed by a prior systematic review (Chaudhry et al.) ⁴ , quantitative reports were considered “hypothesis-testing” if the investigators compared data between groups or across time periods, using statistical tests to assess differences. We further categorized hypothesis-testing studies into 5 study types. RCTs were defined as studies that had a control and experimental arm for which the intervention (health IT) was randomly assigned. Cohort trials were defined as non-randomized studies for which a concurrent control arm was included. Observational studies were most often before-and-after studies in which the “before” group served as the only control. Time series analyses were studies for which time-series statistical analyses were conducted. Lastly, case-control studies were studies for which cases and controls were picked retrospectively, based on exposure to health IT. ⁴

Data Extraction and Analysis

For each article included, both reviewers extracted information regarding patients, clinicians involved, setting, intervention, and outcomes for each of the studies. The safety outcomes evaluated were categorized into the following groups: 1) ADEs or adverse events (AEs); 2) mortality; 3) thrombosis or bleed; 4) length of stay (LOS); 5) infection rates; 6) readmission, admission, or emergency department (ED) visits; 7) fall rates or pressure ulcer; 8) hemodynamic instability or intensive care unit (ICU) transfer; 9) myocardial infarction (MI) or cardiac events; 10) chronic disease exacerbations; and 11) altered mental status (AMS) or stroke incidence.

Adapting methodology used in prior reviews, positive studies were those in which the primary outcome studied showed statistically significant improvement. Negative studies were those for which there were statistically significant worse patient safety outcomes. Mixed or non-significant studies were those for which the primary outcome had a non-significant result but secondary outcomes had a positive result, or studies in which all outcomes were non-significant. ¹⁵ Studies for which patient safety outcomes were not the primary outcome studied were included and the non-patient safety outcome endpoints were not analyzed. Consensus was reached during review discussions. A narrative synthesis method was used to integrate the findings into descriptive summaries. Sub-analyses of positive studies, mixed studies, and randomized controlled trials were conducted.

RESULTS

The search strategy identified 6138 articles. After removal of duplicate articles and articles available only in a foreign language, there were 4736 articles that underwent title review. Based on title alone, 817 (17%) were considered not appropriate for the study. Another 344 articles were added based on a review of the references of the systematic reviews found during the title review process. A total of 4263 articles then underwent abstract review, with 135 articles included for full two-person review. Sixty-eight articles met all of the study inclusion criteria ( Figure 1 ).

Reviewer Agreement

Of the 135 papers reviewed by 2 reviewers, agreement about eligibility for inclusion in the systematic review was excellent 90.4% ( k  = 80.9%; 95% CI, 71.0-90.8%). Of the 69 studies included in the final review, the level of chance-corrected agreement for scientific merit between reviewers was excellent, with a quadratic-weighted k statistic of 88.9% (95% CI, 84.6-93.3%).

Descriptive Analysis of All Studies

Types of health IT and outcomes studied

There was at least one article for every type of health IT pre-identified. More than one health IT was analyzed in 22 studies (31%), and in those cases, all of the health IT tools studied were included in the analysis. The most common health IT interventions were CDS ( n  = 40) and CPOE ( n  = 27) ( Table 3 ). Four health IT tools (electronic medication reconciliation, electronic clinical pathways, patient portal, and smart pumps) were included in only one study, and another 2 tools (HIE and automated medication dispensing) were only found in 2 studies each.

Table 3:

Summary of Findings

Characteristic	Total (%)	Positive Studies	Non-significant or Mixed Results Studies	Negative Studies
Total (%)	69 (100)	25 (36)	43 (62)	1 (1)
Study Design
Randomized Control Trial	18 (26)	5 (7)	13 (19)	–
Cohort	21 (30)	8 (12)	13 (19)	–
Observational	22 (31)	10 (14)	12 (17)	1 (1)
Time Series	7 (10)	2 (3)	5 (7)	–
Case-Control	1 (1)	1 (1)	–	–
Setting
Inpatient	59 (86)	25 (36)	33 (48)	1 (1)
Outpatient	10 (14)	1 (1)	9 (13)	–
Long-Term Care	1 (1)	–	1 (1)	–
Multi-Center	19 (28)	4 (6)	15 (22)	–
Clinicians Affected
Physicians	55 (80)	19 (28)	36 (52)	1 (1)
Nurses	10 (14)	4 (6)	6 (9)	–
Other	5 (7)	2 (3)	3 (4)	–
Pharmacists	2 (3)	1 (1)	1 (1)	–
Country
United States	51 (75)	19 (28)	31 (46)	1 (1)
Non-United States	19 (28)	7 (10)	12 (17)	–
Methodological Quality Assessment Score
0–3	20 (29)	10 (14)	10 (14)	–
4–6	34 (49)	11 (16)	22 (32)	1 (1)
7–10	15 (22)	4 (6)	11 (16)	–
Type of Health IT Intervention Studied
Clinical decision support (CDS)	40 (58)	15 (22)	25 (36)	–
Computerized provider order entry (CPOE)	27 (39)	10 (14)	16 (23)	1 (1)
Automated error detection	4 (6)	2 (3)	2 (3)	–
Electronic medication administration record (eMAR)	4 (6)	1 (1)	3 (4)	–
Electronic health record (EHR)	4 (6)	–	4 (6)	–
Med Administration Barcodes	3 (4)	2 (3)	1 (1)	–
Health information exchange (HIE)	2 (3)	1 (1)	1 (1)	–
Automated dispensing	2 (3)	1 (1)	1 (1)	–
Electronic medication reconciliation	1 (1)	–	1 (1)	–
Electronic Clinical Pathways	1 (1)	–	1 (1)	–
Patient Portal	1 (1)	–	1 (1)	–
Smart pumps	1 (1)	1 (1)	1 (1)	–
No. of IT Applications	1 (1)	–	1 (1)	–
Patient Outcomes Studied
Adverse Drug Events or Adverse Events	37 (53)	17 (25)	20 (29)	–
Mortality	18 (26)	5 (7)	12 (18)	1 (1)
Readmission, admission, or Emergency dept. visits	16 (24)	4 (6)	12 (18)	–
Thrombosis or Bleed	10 (14)	3 (4)	7 (10)	–
Length of Stay	8 (12)	4 (6)	4 (6)	–
Infection Rates	8 (10)	5 (7)	3 (4)	–
Fall Rates	3 (4)	2 (3)	1 (1)	–
Hemodynamic Instability or ICU transfer	3 (4)	1 (1)	2 (3)	–
Myocardial Infarction or Cardiac Events	3 (4)	–	3 (4)	–
Chronic Disease Exacerbations	3 (4)	1 (1)	2 (3)	–
Altered Mental Status or Stroke incidence	2 (3)	–	2 (3)	–
Pressure Ulcers	1 (1)	–	1 (1)

Note: Studies can be counted in more than one category where applicable. Abbreviations: dept. = department; ICU = Intensive care unit. Other category under clinicians refers to either not reported or not applicable study population.

The patient safety outcomes studied varied widely. The most common outcomes studied included: ADEs and adverse events (53% of studies), mortality (26%), thrombosis or bleed (14%), LOS (12%), and infection rates (10%). Secondary outcomes were included in the analysis to capture the broadest number of patient outcomes ( Table 3 ).

Study setting and participants

Most of the studies ( n  = 59, 86%) were performed in inpatient settings. A multicenter study design was employed in 19 (28%) of the studies with the majority (15 of these 19, 79%) of the multicenter trials resulting in non-significant clinical outcomes. Multicenter design was used in 75% (6 of 8) of outpatient studies as compared to only 22% (12 of 55) of inpatient studies. The vast majority (80%) of studies assessed physicians, rather than other healthcare practitioners and most studies were conducted in the United States (75%). There were studies for which authors did not specify the clinicians affected by their health IT, nor was it clear that clinicians were a subject under study from the text. In these cases, reviewers classified the clinicians as “NR,” for not reported ( Table 1 ).

Study quality

The study designs were roughly evenly distributed between RCTs, cohort, and observational design studies, with only a few time-series and case-control designed studies. Study quality was approximately evenly distributed across each grouping of ratings (0–3; 4–6; 7–10) ( Table 3 ). Unlike prior studies, we did not find that there was a significant increase in the quality of studies over time. ^11,17 In terms of quality assessment, the weakest aspects of study design tended to be with regard to randomization and allocation. Specifically, the majority of studies failed to have randomization or even a concurrent control group as part of the study design, and most allocation was done at the patient, rather than unit level. Eighty-one percent of studies received a 1 for blinding of outcomes (which meant objective outcomes were assessed without blinding), and 61% of studies received a 2 for follow up (indicating >90% follow up achieved and reported). Reporting of baseline characteristics was variable and evenly distributed between scores of 0, 1, and 2. Notably, this pattern for quality assessment held true for all studies as well as in sub-analysis of positive vs mixed and negative studies Only 10 (24%) of the non-significant studies enrolled over 1000 patients whereas, 11 (44%) of the studies which found a positive effect of health IT on patient outcomes had enrolled more than 1000 patients ( Table 4 ). Larger studies ( n  > 1000 patients) were also more likely to be conducted more recently than smaller studies.

Table 4:

Analysis of Studies Categorized as Mixed Results Studies

Characteristic	Total Mixed Results Studies (%)	Non-significant Studies (%)	Mixed Studies: some positive results, some non-significant results (%)
Total (%)	43 (100)	24 (56)	19 (44)
Study Design
Randomized Control Trial	13 (30)	9 (21)	4 (9)
Cohort	16 (37)	8 (19)	8 (19)
Observational	14 (33)	7 (16)	7 (16)
Setting
Inpatient	23 (53)	16 (37)	17 (40)
Outpatient	9 (21)	7 (16)	2 (5)
Long-Term Care	1 (2)	1 (2)	–
Multi-Center	15 (35)	9 (21)	6 (14)
Methodological Quality Assessment Score
0–3	10 (23)	6 (14)	4 (9)
4–6	22 (51)	12 (28)	10 (23)
7–10	11 (26)	6 (14)	5 (12)
Type of Health IT Intervention Studied
Clinical decision support (CDS)	26 (60)	15 (36)	11 (26)
Computerized provider order entry (CPOE)	17 (40)	7 (16)	11 (26)
Automated error detection	2 (5)	2 (5)	–
Electronic medication administration record (eMAR)	3 (7)	1 (2)	2 (5)
Electronic health record (EHR)	3 (7)	2 (5)	1 (2)
Medication Administration Barcode	1 (2)	–	1 (2)
Health information exchange (HIE)	1 (2)	1 (2)	–
Automated dispensing	1 (2)	1 (2)	–
Electronic medication reconciliation	1 (2)	1 (2)	–
Patient Portal	1 (2)	1 (2)	–
Smart pumps	1 (2)	1 (2)	–
No. of IT Applications	1 (2)	–	1 (2)
Patient Outcomes Studied
Adverse Drug Events or Adverse Events	18 (42)	9 (21)	9 (21)
Readmission, admission, or Emergency dept. visits	12 (28)	9 (21)	3 (5)
Mortality	10 (23)	7 (17)	3 (5)
Thrombosis or Bleed	9 (21)	6 (14)	4 (9)
Length of Stay	4 (9)	–	4 (9)
Infection Rates	4 (9)	1 (2)	3 (5)
Myocardial Infarction or Cardiac Events	3 (7)	2 (5)	1 (2)
Hemodynamic Instability or ICU transfer	2 (5)	2 (5)	–
Chronic Disease Exacerbations	2 (5)	–	2 (5)
Fall Rates	1 (2)	1 (2)	–
Pressure Ulcers	1 (2)	1 (2)

Abbreviations: dept. = department; ICU = Intensive care unit.

Effects of health IT on patient safety outcomes

Of the 69 studies, the majority ( n  = 43, 63%) had either non-significant findings with respect to patient safety outcomes, or mixed outcomes. Only 25 (36%) studies showed a statistically significant positive effect of health IT on the primary patient safety outcome assessed. There was also 1 (1%) study that found that health IT resulted in an increased mortality rate. There was a significant increase in the number of studies published on health IT and patient safety outcomes over time ( Figure 2 ).

Figure 2:

Number of studies published as a function of publication year. Even though the number of studies published on health information technology (IT) has increased significantly, the number of randomized controlled trials (RTC) published annually has only seen a modest increase over time.

Analysis of Positive Studies

The 25 studies that found that health IT had a positive effect on the primary patient safety outcome were mostly observational trials (40%) or cohort trials (30%). The majority of the positive studies were single center trials ( n  = 20), conducted in the United States ( n  = 19).

The vast majority of studies that found a positive effect of health IT occurred in the inpatient setting ( n  = 24, 96%). There was only one trial demonstrating a positive effect of health IT on patient safety outcomes in the outpatient setting, and none in the long-term care setting. There was no significant difference in the sample sizes or quality score of the positive studies as compared the mixed result or null studies ( Table 3 ).

Positive benefit on patient safety outcomes was demonstrated in studies evaluating CDS, CPOE, HIE, automated error detection, eMAR, medication administration barcodes, automated dispensing, and smart pumps. The health outcomes involved were adverse events ( n  = 16 studies), mortality ( n  = 4), LOS ( n  = 4), readmission rates or ED visits ( n  = 2), prevention or reduction of thrombosis or bleeding ( n  = 2), infection rates ( n  = 2), and rates of pressure ulcers or falls ( n  = 2), AMS or stroke incidence ( n  = 1), and hemodynamic instability or ICU admission ( n  = 1). The patient safety outcomes for which there were more positive studies than mixed studies were LOS, renal impairment, and fall or pressure ulcer rates ( Table 3 ).

In conducting further sub-analysis of the studies characterized as mixed results studies, it was found that more than half of those studies had non-significant findings with respect to all patient safety outcomes. The remaining studies categorized as mixed results had some secondary patient safety outcomes that were positive ( Table 4 ).

In order to determine which types of outcomes were positively affected by which types of health IT, the effective combinations of the two were analyzed. Overall, CDS, CPOE, or CPOE combined with CDS accounted for 73% of the interventions that were successful. The only health outcomes for which those health IT interventions did not constitute the majority was for infection rates, pressure ulcers, or hemodynamic instability or transfer to the ICU ( Table 5 ).

Table 5:

Analysis of the Health IT Found to be Effective in Improving Specific Patient Safety Outcomes

Total studies = 25 + 18 = 43	Patient Outcome
Type of Health IT Intervention	ADEs or AEs	Mortality	Length of Stay	Thrombosis or Bleed	Infection Rates	Readm, adm, or ED visits	Hemodynamic Instability or ICU transfer	Fall Rates	Chronic Disease Exacerbations	AMS or CVA	Pressure Ulcers	Total
CDS	6	3	3	2	2	3		1	1			21
CDS/CPOE	3		3	3						1		10
CPOE	7	2		1								10
AED		1	1			1						3
No. of IT Applications		1			1						1	3
Medication Administr-ation Barcode/eMAR	2						1					3
Smart Pumps							1					1
EHR					1							1
Barcode	1											1
HIE	1											1
AED/CDS	1											1
CDS/EHR/CPOE					1							1
Total	21	7	7	7	5	4	2	1	1	1	1

Abbreviations: ADE = adverse drug event; AE = adverse event; ED = emergency department; Readm = readmission; adm = hospital admission; ICU = intensive care unit; AMS = altered mental status; CVA = stroke; CDS = clinical decision support; CPOE = computerized provider order entry; EHR = electronic health record; HIE = Health information exchange; AutoDisp = automated dispensation of medication; eMAR = electronic medication administration record; AED = automated error detection system.

Subgroup Analysis: RCTs Only

Of all the study types, RCTs had the smallest percentage of studies demonstrating positive effect of health IT on safety outcomes ( n  = 5, 28%), as compared with all other studies ( n  = 20, 40%). Again, as for the entire group of studies, inpatient studies, physician studies, and US studies were all more common among RCTs ( Table 6 ). There was a much smaller increase in the number of RCT studies published over time, as compared to all studies ( Figure 2 ).

Table 6:

RCT Trials

Characteristic	Total (%)	Positive Studies	Non-significant or Mixed Results Studies
Total (%)	18 (100)	5 (28)	13 (72)
Setting
Inpatient	10 (56)	5 (28)	5 (28)
Outpatient	7 (39)	–	7 (39)
Long-Term Care	1 (6)	–	1(6)
Multi-Center	8 (44)	1 (6)	7 (39)
Clinicians Affected
Physicians	17 (94)	4 (22)	13 (72)
Nurses	2 (11)	1 (6)	1 (6)
Other	1 (6)	–	1 (6)
Pharmacists	1 (6)	–	1 (6)
Country
USA	12 (67)	3 (17)	9 (50)
Non-USA	6 (33)	2 (11)	4 (22)
Study Quality
0–3	5 (28)	1 (6)	4 (22)
4–6	9 (50)	3 (17)	6 (33)
7–10	4 (22)	1 (6)	3 (17)
Type of Health IT Intervention
Clinical decision support	14 (78)	4 (22)	10 (56)
Computerized provider order entry	3 (17)	1 (6)	2 (11)
Electronic health record	1 (6)	–	1 (6)
Smart Pumps	1 (6)	1 (6)	–
eMedical Reconciliation	1 (6)	–	1 (6)
Health information exchange (HIE)	1 (6)	–	1 (6)
Patient Portal	1 (6)	–	1 (6)
Patient Outcomes Studied
Mortality	6 (33)	–	6 (33)
Readmission, admission, or Emergency dept. visits	6 (33)	1 (6)	5 (28)
Adverse drug events or adverse events	3 (17)	1 (6)	2 (11)
Chronic Disease Exacerbations	3 (17)	1 (6)	2 (11)
Hemodynamic Instability or intensive care unit transfer	3 (17)	1 (6)	2 (11)
Thrombosis or Bleed	2 (11)	1 (6)	1 (6)
Length of stay	2 (11)	1 (6)	1 (6)
Myocardial infarction or Cardiac Events	2 (11)	–	2 (11)
Infection Rates	1 (6)	–	1 (6)
Fall rates	1 (6)	1 (6)	–
Altered mental status or stroke incidence	1 (6)	–	1 (6)

The quality of the RCTs was significantly higher than the non-RCT studies ( P  < .001). The quality of the RCT studies did not improve over time (mean RTCs before 2003, 6.9 and after 2003, 7.2), unlike previously reported. ^11,17

Most RCTs studied patient mortality and readmission, admission, and ED visits. For these outcomes, only one study found a benefit of health IT ( Table 6 ). ¹⁰²

DISCUSSION

Overall Significance

Our finding that most studies had mixed, rather than positive effects on patient safety outcomes, is consistent with almost all prior systematic reviews conducted on health IT and patient safety outcomes. We also found a paucity of outpatient studies, studies evaluating large numbers of patients, and randomized control trials. Given the national priority placed on adoption and use of health IT, our work highlights the urgent need to better evaluate the use of multiple types of health IT on a variety of patient safety outcomes and in a variety of healthcare settings.

Summary of Findings

Demonstrating the benefit of health IT is challenging for several reasons. First, adverse patient outcomes that can be expected to be modified by the implementation of health IT are generally rare events, necessitating large study samples. ^108,109 We only found 21 studies (31%) that had > 1000 patient study subjects. In addition, randomized control trials evaluating health IT are difficult to conduct, limiting the quality of evidence on this topic. Randomization is generally not feasible within an individual unit or practice, and thus has to be conducted across settings. In addition, health IT is costly to purchase, resource-intensive to implement and typically purchased for an entire practice or institution. Not surprisingly, we found that the rates of randomized control trials assessing the effects of health IT on patient safety outcomes are increasing more slowly than the rate of research on this topic overall ( Figure 2 ).

Consistent with those constraints, in regards to the types of health IT studied, we found that CDS was the most commonly studied health IT intervention. This is likely due to its inherent nature—it is a software-based intervention that can be turned on and turned off, making it well suited for randomized control, before-and-after, or time series designs. Furthermore, because it is software-based it can be trialed at multiple institutions at once; as such, 63% of the multicenter trials focused on CDS. In contrast to CDS, however, many of the individual tools studied had only one or two quantitative publications.

We found only 10 studies conducted in the outpatient setting, despite the fact that the majority of care is given in the outpatient setting. ^4,110 Similarly, we found only 1 study conducted in the long-term care setting. While it has been shown that ambulatory care settings have until recently lagged behind larger institutions in engaging in health IT adoption, ^111,112 given the importance of primary care to population health and prevention, the ambulatory care setting stands to gain a lot from rigorous study of the use of health IT to improve patient safety outcomes.

Future Directions and Policy Implications

Given our findings, this review underscores important future directions for this field of research. First, additional large studies are needed to evaluate the effect of health IT on patient safety outcomes, particularly in the outpatient and long-term care settings. Second, a more uniform system for characterizing health IT tools will be needed to facilitate comparison between studies of health IT interventions. CDS, the most commonly studied health IT, for example, covers a very broad range of actual interventions. Third, as the field continues to develop, more cross-institutional studies and collaborations will be required in order to capture the impact of the newest of the emerging health IT tools, such as patient portals and HIE systems.

From a public policy perspective, discussions are occurring in both the academic community and among regulatory agencies as to how to best regulate health IT. The Federal, Food, Drug, and Cosmetic Act recently declared health IT a medical device under regulatory jurisdiction of the US Food and Drug Administration (FDA). ¹¹³ To date, the FDA has not yet exerted its regulatory authority over the vast majority of health IT tools. The Office of the National Coordinator for Health Information Technology has also published Safety Assurance Factors for EHR Resilience (SAFER) guides designed to help organizations assess and optimize health IT safety. ¹¹⁴ However, given the mixed findings of many research studies on the effects of health IT on patient safety outcomes, and one study demonstrating a hazardous effect, ongoing studies will be critical to ensure patients remain safe and to better determine which types and features of health IT actually improve care for patients.

Limitations

This review has several key limitations. The first is a direct correlate of the quantity and scope of the literature. Despite performing a comprehensive search, only a limited set of articles with quantitative data were identified. For many important types of health IT, only a few studies reporting the impact on actual patient outcomes were found, even among technologies that are being promoted by government policy. As with all systematic reviews, this review also faced the limitations imposed by publication bias, for which studies with positive results are more likely to be published than those with non-significant findings. Proportionally, however, we did find more studies with non-significant findings than not, which would suggest that our findings may be conservative in their estimate of the number of studies for which no significant effect of health IT was found. We also confined our search to English language publications, which may have precluded us from finding additional relevant studies. Given these limitations, it is possible that certain types of technology were underrepresented in this review, such as emerging technologies (like mobile technologies), patient portals, or HIE.

For this review, we chose to use a quality scale that has been previously used and published in measuring the quality of the study of health IT. ^9,11,17 While there are other widely utilized scales that might have been chosen, such as the Cochrane rating system, the scale we utilized has additional bias analysis categories not contained in other scales which we felt made it most rigorous for the quality analysis we were employing. Lastly, there is considerable heterogeneity as to what defines certain types of health IT. For example, CDS has become an umbrella term for many different types of decision support that can be implemented in different ways. We relied on authors’ classifications for health IT tools in determining the type of health IT evaluated, rather than addressing this level of variability. This assumption may have led to an overrepresentation of CDS in the literature.

The authors also recognize that the impact of health IT is greatly influenced by technical, organizational, political, and social factors. Controlling for these in the context of a systematic review is extremely difficult given that authors of the original studies are often not able to measure or quantify these factors, and instead rely on well-matched controls to mitigate these effects. The rating system of study quality is the authors’ attempt to guide readers as to which studies may most effectively control for larger, broader factors.

CONCLUSION

This review has important implications relevant to multiple stakeholders in healthcare, including providers, consumers, policymakers, and vendors. As the nation invests more heavily in health IT, understanding the effects on patient safety outcomes is critical. While there are certain health IT tools that are well studied and are demonstrating safety benefits for patients, there are many areas that are vastly understudied. This review underscores the need for additional, high quality, large-scale studies in multiple settings to better understand how health IT is actually impacting patients. Without such research, we will not be able to identify which health IT tools are indeed effective and in what settings we can expect the greatest benefit.

Author Statements

Funding : This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Competing Interests: The authors have no competing interests to declare.

Author Contribution

S.B.: Study design, data collection, data analysis and interpretation, writing, editing and figure creation, and editing.

R.K.: Study design, data interpretation, writing, editing, and figure editing.

Z.G.: Data collection, data analysis and interpretation, manuscript editing.

C.J.: Data collection, manuscript editing.

I.K.: Data collection, manuscript editing.

R.A.: Study design, literature search, manuscript writing and editing.

D.D.: Study design, literature search, manuscript writing and editing.

E.A.: Study design, data collection, data analysis and interpretation, writing, editing, and figure editing.

SUPPLEMENTARY MATERIAL

Supplementary material is available online at http://jamia.oxfordjournals.org/ .