Characterization of Adverse Events Detected in a Large Health Care Delivery System Using an Enhanced Global Trigger Tool over a Five-Year Interval

Abstract

Objective

To report 5 years of adverse events (AEs) identified using an enhanced Global Trigger Tool (GTT) in a large health care system.

Study Setting

Records from monthly random samples of adults admitted to eight acute care hospitals from 2007 to 2011 with lengths of stay ≥3 days were reviewed.

Study Design

We examined AE incidence overall and by presence on admission, severity, stemming from care provided versus omitted, preventability, and category; and the overlap with commonly used AE-detection systems.

Data Collection

Professional nurse reviewers abstracted 9,017 records using the enhanced GTT, recording triggers and AEs. Medical record/account numbers were matched to identify overlapping voluntary reports or AHRQ Patient Safety Indicators (PSIs).

Principal Findings

Estimated AE rates were as follows: 61.4 AEs/1,000 patient-days, 38.1 AEs/100 discharges, and 32.1 percent of patients with ≥1 AE. Of 1,300 present-on-admission AEs (37.9 percent of total), 78.5 percent showed NCC-MERP level F harm and 87.6 percent were “preventable/possibly preventable.” Of 2,129 hospital-acquired AEs, 63.3 percent had level E harm, 70.8 percent were “preventable/possibly preventable”; the most common category was “surgical/procedural” (40.5 percent). Voluntary reports and PSIs captured <5 percent of encounters with hospital-acquired AEs.

Conclusions

AEs are common and potentially amenable to prevention. GTT-identified AEs are seldom caught by commonly used AE-detection systems.

Since the Institute for Health Care Improvement’s Global Trigger Tool1 (GTT) was developed in 2003, it has been translated into Danish, German, and Swedish; adapted for the United Kingdom; and implemented by hundreds of hospitals in multiple countries as a means of measuring adverse event (AE) rates related to their patient safety efforts (Institute for Healthcare Improvement 2013). A number of studies have evaluated its performance (Classen et al. 2008; Naessens et al. 2010), compared it to alternative strategies to detect adverse events (AEs; Classen et al. 2011; Dolores Menendez et al. 2010; Levinson 2010a; Naessens et al. 2009), and described its adaptation and/or use by hospitals, health care systems, or government entities in various countries (Dolores Menendez et al. 2010; Good et al. 2010; Levinson 2010b; Haraden and Leitch 2011). Recently, it has been used to examine temporal trends in AEs in North Carolina hospitals and a large multihospital system in Florida (Landrigan et al. 2010; Garrett et al. 2013), and to assess incidence of AEs in three large U.S. tertiary care centers (Classen et al. 2011). These reports provided limited information on the nature of the AEs detected and largely focused on AE incidence, severity and preventability. As such, more detailed characterization of AEs, including AE type and source, remained to be explored. Examining rates for different types of AEs (procedure-related, adverse drug events, hospital-acquired infection, etc.) may reveal progress—or the need for progress—in some areas that may be missed when looking only at the total AE rate. Such additional information can be helpful for organizations desiring to use the GTT to guide quality improvement efforts.

We have previously described the adaptations we made to the GTT review process and data collection to provide the detailed information necessary to understand the kind of harm our patients are experiencing—as a first step toward identifying effective mechanisms for improvement—at a sustainable cost for an ongoing measurement system (~$2 per admitted patient within the target population; Good et al. 2010; Kennerly et al. 2013). Here, we report the detailed characteristics of AEs identified through 5 years of use of a refined GTT method in the acute care hospitals of a large and diverse health care system in the United States.

Methods

Setting

Baylor Health Care System (BHCS) is an integrated health care delivery system in the Dallas–Fort Worth metroplex, serving patients in North Texas and beyond, and comprising 14 owned, leased, and affiliated hospitals and >190 primary, specialty, and senior health care ambulatory centers. AE data are presented here for the eight general acute care hospitals that have been part of BHCS since January 1, 2007, ranging in size and location from an urban 1,026-bed tertiary care hospital (~40,000 admissions per year), to a 69-bed community hospital (~4,000 admissions per year) outside the urban metroplex. BHCS implemented and adapted the IHI’s GTT methodology in 2006 (Good et al. 2010; Kennerly et al. 2013) and has used it to monitor patient harm in its acute care hospitals since that time.

Patient Record Sampling

The patient record sampling strategy has been previously described (Kennerly et al. 2013). Briefly, patient records were eligible for inclusion if the patient was formally admitted to a BHCS acute care hospital between January 2007 and December 2011, had a length of stay ≥3 days, and was ≥18 years of age, and the record was closed and completed. BHCS policy requires completion of the discharge summary and all coding within 30 days post-discharge; as the GTT sampling is retrospective, a “buffer” period was built in, considering only charts for which ≥60 days had elapsed since discharge. Patients admitted for hospice or rehabilitation, or for psychiatric or addictive diseases were excluded. A random sample of eligible patient records was drawn monthly from each hospital: for small hospitals, a minimum of 10 records were reviewed; for midsize hospitals, a set number corresponding to 2 percent of the recent discharge volume for that hospital; and for the 1,026-bed hospital, 35 records. This distribution was intended to review sufficient charts at the small hospitals to allow meaningful assessments of patient safety on a quarterly to annual basis; at intermediate hospitals, monthly to quarterly; and at our largest hospital, to review no more charts than necessary to get an accurate monthly estimate, while taking into account the diversity of care it provides (Kennerly et al. 2013).

Data Collection and Analysis

The process for reviewing patient records has been previously described in detail (Good et al. 2010). Professional nurse reviewers entered AE data into a Microsoft Access workflow and data collection tool developed for this project. Inter-rater reliability was tested and found satisfactory (κ = 0.62; Kennerly et al. 2013). Data for each identified AE were collected relating to its presence on admission, severity (described using the NCC MERP Index categories E [temporary harm that required intervention], F [temporary harm that required initial or prolonged hospitalization], G [permanent harm], H [intervention required to sustain life], and I [death] [Griffin and Resar 2007]), type (AE due to care provided vs. indicated care omitted), preventability (preventable, probably preventable, possibly preventable, not preventable, unable to determine), AE category (listed below in the Results section), and a brief narrative description of the AE using the Situation-Background-Assessment (S-B-A) format. A 20-minute time limit per patient record was imposed. Confirmatory review of each AE report, based on the narrative description of the AE rather than an independent review of the record, was conducted by members of the BHCS Office of Patient Safety, including the GTT project manager, senior nurses with leadership responsibilities in Patient Safety, and, when needed, a physician consultant. Table S1 compares and contrasts the IHI GTT and the BHCS adaptation.

We examined absolute numbers of AEs (present on admission and hospital acquired) detected in the GTT sample from January 2007–December 2011, as well as their breakdown by severity of harm, preventability, and AE category. The AE categories were developed from a combination of expert opinion, feedback from reviewers, and analysis of the narrative descriptions of the identified AEs in the 2006–2007 review using SPSS Text Analysis for Surveys 2.0 (SPSS, Inc., Chicago, IL). This process for developing and defining AE categories did not produce a stable taxonomy until January 2008. Therefore, AE category data are only reported for the period January 2008–December 2011.

We determined the AE rate as AEs per 1,000 patient-days, AEs per 100 admissions, and percentage of admissions with ≥1 AE, and, for our system-level rates, used both the raw data and a “weighted calculation” (weighting the contributions of each of the eight included hospitals based on the relative facility size with respect to the population that was eligible to be sampled) to account for our data collection process’s relative oversampling of small hospitals and undersampling of the largest hospital (Kennerly et al. 2013).

A chi-square test was performed to determine whether there was an association between severity and preventability among all hospital-acquired AEs.

By matching medical record and account numbers, we determined what percentage of patient encounters identified by the GTT as having a hospital-acquired AE also appeared in the electronic voluntary incident-reporting system, used throughout BHCS for frontline staff to report AEs or near misses, or produced a positive AHRQ Patient Safety Indicator (PSI) flag, based on the March 2012 version of the AHRQ PSI software (Agency for Healthcare Research and Quality, 2012), on review of administrative data. This comparison was limited to patients discharged between October 2008 and December 2011, as the PSI data prior to October 2008 did not differentiate between “present on admission” and “hospital acquired.”

All analyses were conducted using SAS 9.2 (Cary, NC).

This project was approved by the Baylor Research Institute IRB.

Results

From January 2007 to December 2011 there were 643,744 admissions to the eight BHCS acute care hospitals. Of these, 98,004 (15.2 percent) were patients <18 years, 261,849 (40.6 percent) lasted <3 days, and 1,207 (0.2 percent) were for hospice, rehabilitation, or for psychiatric or addictive diseases. This left 346,696 (53.9 percent) admissions eligible for sampling. We reviewed records from 9,017 patient encounters and identified 3,430 AEs. Table 1 shows the breakdown of these AEs by presence on admission versus hospital acquired, by care provided versus indicated care omitted (for hospital acquired only), and by severity and preventability. Of the identified AEs, 1,300 (37.9 percent) were present on admission. Looking at the present-on-admission and hospital-acquired AEs together, AEs most commonly had an NCC MERP severity level of F (1,625/3,430 [47.4 percent]) and were possibly preventable (2,050/3,430 [59.2 percent]). Among the present-on-admission AEs, the predominance of level F harm was more marked at 1,021/1,300 (78.5 percent); present-on-admission AEs were also more likely to be at least possibly preventable (1,139/1,300 [87.6 percent] vs. 2,648/3,430 [77.2 percent] overall). The vast majority of hospital-acquired AEs stemmed from care provided (1,854/2,129 [87.1 percent]), and level E harm predominated (1,348/2,129 [63.3 percent]). Preventability differed greatly between the hospital-acquired AEs stemming from care provided versus from indicated care omitted: 62/116 (53.5 percent) of AEs from indicated care being omitted were probably or definitely preventable, compared to only 220/1,854 (11.9 percent) of AEs stemming from care provided.

Table 1.

Severity and Preventability of Adverse Events Identified Using the Global Trigger Tool across Eight Baylor Health Care System Acute Care Hospitals (January 2007–December 2011)

			Hospital Acquired
	Total*	Present on Admission	Care Provided	Indicated Care Omitted	Unable to Determine
Severity†
E	1,572	223	1,194	70	84
F	1,625	1,021	516	37	51
G	33	14	16	0	3
H	153	28	106	7	12
I	47	14	22	2	9
Preventability‡
Preventable	68	23	30	14	1
Probably preventable	550	306	190	48	5
Possibly preventable	2,030	810	1,098	52	70
Not preventable	675	113	503	2	57
Unable to determine	107	48	33	0	26
Total	3,430	1,300	1,854	116	159

^*Includes one AE for which it could not be determined if it was “hospital acquired” or “present on admission.”

^†NCC MERP classifications: E = temporary harm that required intervention; F = temporary harm that required initial or prolonged hospitalization; G = permanent harm; H = intervention required to sustain life; I = death.

^‡Based on the reviewer’s clinical knowledge, he/she feels the AE was as follows: definitely preventable (Preventable); more likely than not the AE could have been prevented (Probably Preventable); there is some chance the AE could have been prevented (Possibly Preventable); the event was definitely not preventable (Not Preventable); not able to be assigned to any of the categories of preventability (Unable to Determine).

GTT-derived AE rates for the eight acute care hospitals combined are shown in Table 2. Overall, we observed an AE rate of 61.4 AEs/1,000 patient days, or 38.1 AEs/100 discharges. Put in the patient’s perspective, an estimated 32.1 percent of patients discharged between January 2007 and December 2011 who met the eligibility criteria listed above had ≥1 AE. Looking at hospital-acquired and present-on-admission AEs separately, rates of both were substantial, although hospital-acquired rates were greater. Table 2 also shows the “weighted” AE rates (i.e., corrected for the sampling bias), which were very similar to the “raw” estimates.

Table 2.

Adverse Event (AE) Rates for the Baylor Health Care System (BHCS) Eight Acute Care Hospitals (January 2007–December 2011)

	Total*			Hospital Acquired			Present on Admission
	AEs/1,000 Pt-Days	AEs/100 Discharges	≥1 AE (%)	AEs/1,000 Pt-Days	AEs/100 Discharges	≥1 AE (%)	AEs/1,000 Pt-Days	AEs/100 Discharges	≥1 AE (%)
BHCS	61.4	38.1	32.1	38.1	23.6	20.7	23.3	14.4	12.9
BHCS (weighted)†	59.6	38.9	32.5	37.4	24.6	21.3	22.2	14.3	12.7

Note. Weighted values were based on the relative volume of patients meeting the inclusion criteria.

^*Records from 9,017 patient encounters were reviewed from a total of 315,274 eligible encounters. Those records involved a total of 55,916 patient-days, from a total of 2,168,956 patient-days for admissions meeting the inclusion criteria.

^†Weighted according to the contributions of each of the eight acute care hospitals, based on the relative facility size with respect to the population that was eligible to be sampled, to account for our data collection process’s relative oversampling of small hospitals and undersampling of the largest hospital (Kennerly et al. 2013).

Table 3 (and Table S2) shows the preventability of hospital-acquired AEs by severity. In all severity categories the majority of AEs were judged to be possibly preventable. Only 7/45 (15.6 percent) of the AEs judged definitely preventable were of severity level G or above. However, a further 22/243 (9.1 percent) AEs judged probably preventable fell in these severity levels. The test for interaction between preventability and severity was highly significant (p < .0001), indicating these characteristics of the AEs are not independent. However, no clear pattern of how they are related (e.g., preventability increasing with severity) was observed.

Table 3.

Severity and Preventability of Hospital-Acquired Adverse Events Identified Using the Global Trigger Tool across the Eight Baylor Health Care System Acute Care Hospitals

		Preventability†
Severity*	Total	Yes	Probably	Possibly	No	Cannot Tell
E	1,348	26	146	750	395	31
F	604	12	75	358	145	14
G	19	1	0	12	4	2
H	125	6	17	85	10	7
I	33	0	5	15	8	5
Total	2,129	45	243	1,220	562	59

^*NCC MERP classifications: E = temporary harm that required intervention; F = temporary harm that required initial or prolonged hospitalization; G = permanent harm; H = intervention required to sustain life; I = death.

^†Based on the reviewer’s clinical knowledge, he/she feels the AE was as follows: definitely preventable (Preventable); more likely than not the AE could have been prevented (Probably Preventable); there is some chance the AE could have been prevented (Possibly Preventable); the event was definitely not preventable (Not Preventable); not able to be assigned to any of the categories of preventability (Unable to Determine).

Table 4 (and Table S3) shows an analysis of identified hospital-acquired AEs by category during the most recent 4-year interval. The overwhelming majority of AEs fell within one of three categories: surgical/procedural AE (676/1,669 [40.5 percent]), medication AE (435/1,669 [26.1 percent]), and hospital-acquired infection (212/1,669 [12.7 percent]). The same three categories accounted for the most severe AEs—NCC MERP level G (permanent patient harm), H (required intervention to sustain life), or I (contributed to or resulted in death): surgical/procedural (46/132 [34.8 percent]), medication (38/132 [28.8 percent]), and hospital-acquired infection (16/132 [12.1 percent]). Of these high severity categories, infection was the most likely to be preventable (86/212 [40.6 percent] of infection AEs were either preventable or probably preventable). While the surgical/procedural and medication AEs were far less likely to be judged preventable or probably preventable (15/676 [2.2 percent] and 18/435 [4.1 percent], respectively), substantial portions of these categories were neverthetheless judged possibly preventable (490/676 [72.5 percent] of surgical/procedural and 174/435 [40.0 percent] of medication AEs). When considering the “low hanging fruit” of preventable/probably preventable AEs to target for quality improvement purposes, infections and pressure ulcers appear to be the primary targets, accounting for 86/199 (43.2 percent) and 25/199 (12.6 percent) of such events, respectively. Notably, 25/26 (96.2 percent) of the pressure ulcers identified were judged to be probably preventable.

Table 4.

Severity and Preventability of Hospital-Acquired Adverse Events (AEs) Identified Using the Global Trigger Tool across the Eight Baylor Health Care System Acute Care Hospitals by AE Category (January 2008–December 2011^*)

		Severity†					Preventability‡
AE Category	Total	E	F	G	H	I	Yes	Probably	Possibly	No	Cannot Tell
Surgical/procedural AE	676	402	228	7	36	3	4	11	490	169	2
Medication AE	435	312	85	1	34	3	5	13	174	242	1
Infection	212	114	82	0	10	6	2	84	119	7	0
IV related	70	70	0	0	0	0	0	2	67	1	0
Fluid overload/pulmonary edema	51	23	22	0	6	0	1	9	41	0	0
Perinatal AE	45	33	11	0	1	0	0	11	29	5	0
Thrombosis/Embolism	41	17	24	0	0	0	1	4	17	19	0
Sepsis	41	7	20	0	4	10	0	23	16	2	0
Pressure ulcer	26	25	1	0	0	0	0	25	1	0	0
Fall with injury	10	7	2	1	0	0	0	0	9	1	0
Blood transfusion reaction	9	7	1	0	1	0	0	0	2	7	0
Pneumothorax	9	1	7	0	1	0	0	0	9	0	0
Stroke	7	0	4	3	0	0	0	0	3	4	0
Other AE	37	22	10	1	4	0	0	4	21	9	3
Total	1,669	1,040	497	13	97	22	13	186	998	466	6

^*January–December 2007 data were excluded as the list of categories was still under development.

^†NCC MERP classifications: E = temporary harm that required intervention; F = temporary harm that required initial or prolonged hospitalization; G = permanent harm; H = intervention required to sustain life; I = death.

^‡Based on the reviewer’s clinical knowledge, he/she feels the AE was as follows: definitely preventable (Preventable); more likely than not the AE could have been prevented (Probably Preventable); there is some chance the AE could have been prevented (Possibly Preventable); the event was definitely not preventable (Not Preventable); not able to be assigned to any of the categories of preventability (Unable to Determine).

Table 5 (and Table S4) provides additional detail on hospital-acquired AEs that fell within the surgical/procedural category (the most common AE category). The most common type of AE within this category was bleeding (241/676 [35.7 percent]), and the vast majority of the bleeding events were associated with level E harm (temporary, requiring an intervention; 190/241 [78.8 percent]) and were judged to be possibly preventable (237/241 [98.3 percent]). However, this subcategory also accounted for the single largest proportion (13/36 [36.1 percent]) of events associated with level H harm (intervention to sustain life required). Similar detail is provided for subtypes of medication-related AEs in an online appendix (Table S5).

Table 5.

Detail on Hospital-Acquired Adverse Events (AEs) in the Surgical/Procedural Category Identified Using the Global Trigger Tool across the Eight Baylor Health Care System Acute Care Hospitals by AE Category (January 2008–December 2011^*)

		Severity†					Preventability‡
AE Subcategory	Total	E	F	G	H	I	Yes	Probably	Possibly	No	Cannot Tell
Bleeding	241	190	37	0	13	1	0	1	237	3	0
Ileus	75	31	44	0	0	0	0	0	14	61	0
Dysrhythmia	53	29	20	0	4	0	0	2	3	48	0
Laceration of organ/blood vessel	48	29	17	1	1	0	0	4	44	0	0
Urinary retention	40	31	9	0	0	0	1	0	2	37	0
Atelectasis	27	17	9	0	1	0	0	0	27	0	0
Pericardial/pleural effusion	23	8	13	0	2	0	0	0	23	0	0
Hematoma	17	6	8	1	2	0	0	1	15	1	0
Renal insufficiency/failure	9	3	5	0	0	1	0	0	8	1	0
Myocardial infarction	4	0	2	0	2	0	0	0	3	1	0
Intestinal obstruction	3	0	3	0	0	0	0	0	3	0	0
Ischemia/occlusion of blood vessel	3	0	3	0	0	0	0	0	3	0	0
Dehiscence	2	0	1	1	0	0	0	0	2	0	0
Return to surgery	1	0	1	0	0	0	0	0	0	1	0
Other	130	58	56	4	11	1	3	3	106	16	2
Total	676	402	228	7	36	3	4	11	490	169	2

^*January–December 2007 data were excluded as the list of AE categories was still under development during this period.

^†NCC MERP classifications: E = temporary harm that required intervention; F = temporary harm that required initial or prolonged hospitalization; G = permanent harm; H = intervention required to sustain life; I = death.

^‡Based on the reviewer’s clinical knowledge, he/she feels the AE was as follows: definitely preventable (Preventable); more likely than not the AE could have been prevented (Probably Preventable); there is some chance the AE could have been prevented (Possibly Preventable); the event was definitely not preventable (Not Preventable); not able to be assigned to any of the categories of preventability (Unable to Determine).

Few of the hospital-acquired AEs identified by the GTT were also detected by the BHCS electronic voluntary reporting system or by applying the AHRQ PSI criteria to administrative data. Of the 1,186 patient encounters for which the GTT identified ≥1 hospital-acquired AE for the period October 2008–December 2011, only 42 (3.5 percent) also had an AE recorded in the voluntary reporting system, and only 18 (1.5 percent) had a hospital-acquired PSI detected through analysis of administrative data (details by harm score are shown in Table S6).

Discussion

We report the absolute numbers and rates of AEs identified by our modified application of the GTT in a large health care system in the United States over a 4-year period. Our results substantiate the recent observation that AEs are more common than expected (Classen et al. 2011) and show that there is substantial opportunity to reduce risk of harm to patients.

Our observed total AE rates of 61.4 AEs/1,000 patient-days, 38.1 AEs/100 discharges, and 32.1 percent of patients with ≥1 AE were similar to those reported for the Mayo Clinic hospitals in 2005–2007 (23.1–37.9 percent of patients experiencing ≥1 AE; Naessens et al. 2010), lower than those reported using data from October 2004 for three “large US tertiary care centers that had well-established operational patient safety programs” (91 AEs/1,000 patient-days, 49 AEs/100 admissions, and 33.2 percent of patients with ≥1 AE; Classen et al. 2011), higher than the rates reported for a random sample of discharges from 10 North Carolina hospitals from 2002 to 2007 (56.6 AEs/1,000 patient-days, 25.1 AEs/100 admissions, and 18.1 percent of patients with ≥1 AE; Landrigan et al. 2010), and—depending on the rate measure—lower, similar to, and higher than the results reported by Adventist Health System (AHS) for 2009–2011 (85 AEs/1,000 patient days, 38 AEs/100 admissions, and 26 percent of patients with ≥1 AE; Garrett et al. 2013). Like our study, all these reports included AEs that required treatment but did not extend the length of stay (Landrigan et al. 2010; Naessens et al. 2010; Classen et al. 2011; Garrett et al. 2013), but they did not include AEs judged to be the result of failure to provide indicated care (which made up 5.4 percent of the hospital-acquired AEs in our data). Additionally, we have previously shown that our inclusion criterion of ≥3 days LOS (compared to the ≥24 hours LOS used in other studies) results in an approximately 1.3-fold “inflation” of the AEs/100 admissions and the percentage of patients with ≥1 AE (the number of AEs/1,000 patient days was minimally impacted with a 1.02-fold “inflation”; Kennerly et al. 2013). Adjusting for this sampling bias our ≥3 days LOS criterion creates, our rates compare more favorably to previous reports: 60.1 AEs/1,000 patient days, 29.2 AEs/100 admissions, and 24.6 percent of patients with ≥1 AE.

Only the North Carolina study reported hospital-acquired and present-on-admission AE rates separately. Their hospital-acquired AE rate (20.7 per 100 admissions) was very similar to ours (23.6 per 100 admissions), but they observed a substantially lower rate of present-on-admission AEs (4.4 compared to 14.1 per 100 admissions; Landrigan et al. 2010). This difference is important, as our results suggest that reducing AEs due to antecedent care is an important area of focus for improving patient safety in the continuum of care.

Among the 2,129 hospital-acquired AEs we identified, 1,854 (87.1 percent) were due to “care provided”—a result markedly different from a previous report in the literature that found more omission-related events in the medical services (92/162 [57.1 percent] of AEs) and a roughly equal balance in the surgical services (omission-related events accounting for 94/185 [50.8 percent] of AEs; Baker et al. 2004). The majority of the hospital-acquired AEs we identified also had level E harm, and they were at least possibly preventable, although the hospital-acquired AEs judged to be due to “indicated care omitted” or “unable to determine” had higher rates of level F or above harm. Looking at all hospital-acquired and present-on-admission AEs, the proportion of AEs with high levels of harm (G or above) was still low compared to that found by the Department of Health and Human Services Office of the Inspector General (OIG) in a sample of Medicare beneficiaries discharged from acute care hospitals in October 2008 (Levinson 2010b). The OIG, looking only at AEs with at least level F harm, found 38 percent (49/128) of AEs had a severity level of G or above (Levinson 2010b), while our results (when excluding the level E AEs) show only 12.5 percent (233/1,858). It is important to note, however, that the OIG aggregated the findings of GTT review and analysis of administrative data indicators for risk of an AE, did not apply a 20-minute per patient record time limit, and had a patient population limited to Medicare beneficiaries—all factors which could impact the nature and severity of the events detected. Our observed 233/3,430 (6.8 percent) of AEs (hospital acquired and present on admission, all severity levels) with harm of level G or above was very similar to the result reported by Mayo Clinic hospitals in 2005–2007 (25/307 [8.1 percent] patients identified as experiencing an AE suffered G or above harm; Naessens et al. 2010).

Previous studies examining preventability have judged larger proportions of the AEs as “preventable” than the 18.0 percent (618/3,430) considered preventable or probably preventable in our data. The OIG study identified 133/302 (44 percent) of AEs as “clearly” or “likely” preventable(Levinson 2010b), and the North Carolina study identified 364/588 (63.1 percent) of AEs as “preventable”—but did not specify whether this included possibly/probably, in addition to definitely preventable AEs (Landrigan et al. 2010). A systematic review of medical record review-based AE studies found the median preventability to be 43 percent (de Vries et al. 2008), but comparisons on this AE characteristic are made challenging both by the inconsistent scales of preventability used between studies and the subjective nature of the determination. It seems likely that nurse reviewers (on which our method primarily relies) are less comfortable using the judgment of “probably preventable” than “possibly preventable,” especially for complex patient situations. This may explain some of the difference between our results and those of organizations like the OIG, where a physician was always available for consultation during the chart reviews, which may have shifted more preventability judgments toward the more definite categories.

The long-term, routine use of the GTT in BHCS acute care hospitals provided us with a two- to four-fold larger sample and pool of AEs than previous studies (Landrigan et al. 2010; Levinson 2010b; Naessens et al. 2010; Classen et al. 2011; Garrett et al. 2013), which, in turn, provided an opportunity to characterize the AEs in greater detail. The most common types of hospital-acquired AEs we observed were surgical/procedural, medication-related, and infection, while the most preventable were pressure ulcer and sepsis, and those most likely to have severe harm (level G or above) were stroke, sepsis, and pneumothorax.

While the OIG report looked only at “type” of AE for those with temporary harm and did not separate AEs into present on admission versus hospital acquired, it provides some point of comparison—and it is immediately obvious that there is wide divergence, even in the categories used. The most common category of temporary harm (NCC MERP severity E) AE identified by the OIG was medication-related at 73/174 (42 percent) AEs—a category which accounted for only 30.0 percent (312/1,040) of hospital-acquired level E AEs in the BHCS data. Conversely, the most common AE type in our results (surgical/procedural, accounting for 402/1,040 [38.7 percent] of level E hospital-acquired AEs) accounted for only 32/174 (18 percent) of the “temporary harm” AEs identified by the OIG; and our third most-common AE type (infection, accounting for 114/1,040 [11.0 percent] of level E hospital-acquired AEs) made up only 6/174 (4 percent) of the OIG’s “temporary harm” AEs (Levinson 2010b). Similarly, when AEs of all harm levels were considered, surgical/procedural AEs were the most common in our data (676/1,669 [40.5 percent]), while AHS reported that four of their five most frequently found categories were medication related (Garrett et al. 2013). However, consistent with our observations, other AE literature shows surgical/procedural AEs are often the most commonly identified. A systematic review of eight medical-record review-based studies found surgery-related AEs were most common, accounting for a median of 39.6 percent of all identified AEs (de Vries et al. 2008), which is very close to the 40.5 percent we observed. Similarly, a Canadian study found surgery-related AEs accounted for 123/360 (34.2 percent) of AEs (Baker et al. 2004). When the records of patients discharged in 2005 from the three Mayo Clinic Rochester hospitals were examined for the presence of hospital-acquired AHRQ PSIs, 1,082/1,596 (68.7 percent) of those identified were for postoperative or procedure-related conditions (Naessens et al. 2009). Although the explanation for the Mayo PSI results—that AEs from surgery are more amenable to ICD-9-CM coding (Rosen et al. 2005; Naessens et al. 2009)—should not influence the GTT, a similar phenomenon regarding documentation of medical versus surgical AEs may be at work.

As has been noted in other settings, there was little overlap between the AEs identified using the GTT and those reported in the BHCS electronic voluntary reporting system or detected by the AHRQ PSIs. Mayo Rochester found that only 11/65 (17 percent) patients with AEs identified by the GTT were also identified by either a provider report or the application of the AHRQ PSI definitions (Naessens et al. 2009), compared to our findings of 3.5 percent of hospital encounters having ≥1 GTT-identified AEs also having a voluntary AE report record, and 1.5 percent also showing positive for a PSI. This small overlap suggests that these three methods serve different purposes within health care organizations’ operations and should be viewed as complementary rather than interchangeable. Furthermore, the information we obtained through the GTT process on the type of harm patients were experiencing and within which areas of care revealed how limited a view of patient safety administrative data-based measures—such as the AHRQ PSIs and the Medicare Hospital-Acquired Conditions—provide.

Our study has several potential limitations that should be kept in mind. First, the BHCS rates and distribution of AE characteristics may not be generalizable to systems composed of different types of hospitals—or to hospitals in other locations if there is geographic variation in AE patterns. Second, our patient-inclusion criteria for the GTT sample required a minimum length of stay of 3 days, rather than the IHI’s 24-hour requirement. This change eliminated ~40 percent of patient admissions from possible inclusion, which, based on our analysis of admissions to the BHCS acute care hospitals from January 2008 to June 2010, biases our sample against detecting obstetrical and short-stay surgical AEs, and, because substantially more AEs are detected in patient encounters with length of stay ≥3 days, tends to increase our observed AE rates (Kennerly et al. 2013). Other limitations pertain to the tool itself, particularly the subjective nature of judgments regarding whether an AE occurred and its preventability.

While our results provide insight into the types and nature of AEs that occur in hospital settings, much work remains to exploit the full potential of the GTT as a patient safety monitoring and learning tool. For example, we and others are exploring automation of trigger identification in electronic health record-derived data, to enable 100 percent record review—although this work faces challenges related to the need to retrofit the electronic health record systems currently in use that do not incorporate the technology to systematically screen for and help measure AEs (Jha and Classen 2011). Additionally, there would be great value in time trend data and between-hospital comparisons from the perspective of evaluating the effectiveness of patient safety interventions. However, such work is complex, requiring the development of risk adjustment models that take into account changes over time/between-hospital differences in the patient population’s risk profile and the services offered. We have begun developing such a model but cannot yet present meaningful comparisons over time or between the hospitals in our health care system. Furthermore, a recent assessment of the GTT’s ability to detect changes in AE rates over time revealed a wide measurement error range, within which changes in the AE rate cannot be distinguished from normal variation (Mattsson et al. 2013). These factors play an important role in determining how we have used GTT data within our health care system to date. Rather than considering it a monitoring tool, with performance rates reported at regular intervals to hospital leaders and frontline clinicians, we have used the GTT data the same way it is presented here: in aggregate, with an initial emphasis on raising awareness of the frequency with which AEs occur and the importance of addressing this aspect of the quality of care provided, and, as additional data enabled us to delve deeper into the categorization and characterization of the AEs, to guide the priorities of our patient safety improvement work.

When we conducted the initial GTT review within our health care system in 2008 and reported the results to executive and clinical leaders, there was great surprise at the AE rates observed within our hospitals: the GTT data suggested it was approximately 16-fold greater than the estimates based on the voluntary reporting system on which our hospitals had historically relied. These data played an important role in solidifying the commitment of leaders to investing in patient safety in the form of establishing a robust organization-wide patient safety program (Kennerly et al. 2011). There were, however, skeptics, particularly among the clinicians, who needed to be convinced that the GTT-identified events were true AEs. To validate our findings, we provided the details on each of the AEs identified in the initial review to the relevant hospital’s AE Reduction Team (Kennerly et al. 2013). These teams of clinicians agreed that the events were indeed AEs, but, in the absence of clear preventability for many of the AEs, regarded them as not unexpected and therefore not signifying suboptimal care. This response revealed the need for a substantial shift in perspective surrounding patient safety, away from “Did the clinician provide the requisite standard of care?” and toward “Would the patient regard this as an adverse outcome?” In conjunction with growing emphasis on patient-centeredness in our health care system’s quality improvement efforts at this time (Nuss 2013), we engaged in discussions with the medical executive leaders throughout our health care system, asking them to renew their commitment to taking the patient’s perspective to safety—that is, that all risk of harm needs to be minimized, whether the specific instances of AEs are clearly preventable or not.

As additional years’ data enabled us to meaningfully examine the kinds of AEs occurring in our hospitals, and within which areas of care, one of the most important findings was that the “surgical/procedural” was the most common AE category. In recent years, national patient safety initiatives have placed heavy emphasis on adverse drug events and hospital-acquired infections (2012; Rosenthal 2007; U.S. Department of Health and Human Services, n.d.), a focus which was reflected in our organizational priorities for patient safety. The GTT findings created impetus for improvement efforts that targeted surgical care, across the domains of processes of care, organizational culture, and technology. These included a systemwide focus on the Surgical Care Improvement Project Core Measures performance and implementation of the World Health Organization’s Safe Surgery Saves Lives Checklist (Franklin 2013), together with anonymous surveys of operating room nurses every 6–9 months asking how frequently the Checklist was used in a way they would want for themselves or their families. In addition, it prompted us to collaborate with Kaiser Permanente on work related to their high-reliability surgery system (The Commonwealth Fund 2005), specifically to the surgeon’s role in setting a safety-focused tone in the operating room, and has prompted an exploratory study examining the relationship between consistency of the operating room team (surgeon, scrub nurse, and circulating nurse) and patient outcomes—the preliminary results of which show a positive association. Without the local data showing our specific areas with opportunities for improvement, our focus would likely have remained centered on topics which have received greater attention on the national agenda and in the literature. Our use of the GTT is, therefore, helping us better apply our resources to improvement efforts that will benefit our patients most.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Table S1: Comparison of the Institute for Healthcare Improvement (IHI) and Baylor Health Care System (BHCS) Global Trigger Tool Review Process and Data Collection.

Table S2: Severity and Preventability of Hospital-Acquired Adverse Events Identified Using the Global Trigger Tool across the Eight Baylor Health Care System Acute Care Hospitals (January 2007–December 2011).

Table S3: Severity and Preventability of Hospital-Acquired Adverse Events (AEs) Identified Using the Global Trigger Tool across the Eight Baylor Health Care System Acute Care Hospitals by AE Category (January 2008–December 2011*).

Table S4: Detail on Hospital-Acquired Adverse Events (AEs) in the Surgical/Procedural Category Identified Using the Global Trigger Tool across the Eight Baylor Health Care System Acute Care Hospitals by AE Category (January 2008–December 2011*).

Table S5: Detail on Hospital-Acquired Adverse Events (AEs) in the Medication-Related Category Identified Using the Global Trigger Tool across the Eight Baylor Health Care System Acute Care Hospitals by AE Category (January 2008–December 2011*).

Table S6: Patient Encounters with GTT-Identified Hospital-Acquired Adverse Events That Also Had Events Captured by Voluntary Reporting/AHRQ PSIs (October 2008–December 2011).