COMPARISON OF A VOLUNTARY SAFETY REPORTING SYSTEM TO A GLOBAL TRIGGER TOOL FOR IDENTIFYING ADVERSE EVENTS IN AN ONCOLOGY POPULATION

Abstract

Objectives.

There is a lack of research on adverse event (AE) detection in oncology patients, despite the propensity for iatrogenic harm. Two common methods include voluntary safety reporting (VSR) and chart review tools, such as the Institute for Healthcare Improvement’s Global Trigger Tool (GTT). Our objective was to compare frequency and type of AEs detected by a modified GTT compared to VSR for identifying AEs in oncology patients in a larger clinical trial.

Methods.

Patients across six oncology units (from July 1,2013 through May 29,2015) were randomly selected. Retrospective chart reviews were conducted by a team of nurses and physicians to identify AEs using the GTT. The VSR system was queried by Department of Quality and Safety of the hospital. AE frequencies, type, and harm code for both methods were compared.

Results.

The modified GTT detected 0.90 AEs per patient (79 AEs in 88 patients) [95% (0.71, 1.12) AEs per patient] that were predominantly medication AEs (53/79); over half of AEs caused harm to patients (41/79; 52%), but only one quarter were preventable ((21/79; 27%). VSR detected 0.24 AEs per patient (21 AEs in 88 patients) [95% (0.15, 0.37) AEs per patient], a large plurality of which were medication/intravenous related (8/21); over half did not cause harm (70%). Only 2% of AEs (2/100) were detected by both methods.

Conclusions.

Neither the modified GTT nor the VSR system alone are sufficient for detecting AEs in oncology patient populations. Further studies exploring methods such as automated AE detection from electronic health records and leveraging patient-reported AEs are needed.

INTRODUCTION

Due to the nature of their illness, oncology patients are at an increased risk of adverse events (AEs)^{1 2} and are more likely to experience iatrogenic harm than other types of medical patients.³ This risk of harm independently increases with age, length of stay, surgery, and emergency services.⁴ Factors that place oncology patients at increased risk for AEs are the toxic treatments that they receive and need for care coordination among multiple medical professionals.⁵ Lipczak et. al. have identified hazards specific to cancer care, specifically the combination of surgery, radiotherapy, and chemotherapy for treatment purposes.⁶

Monitoring of AEs in oncology has been done in two main ways: through voluntary safety reporting (VSR) by clinicians, and chart review using tools like the Institute for Healthcare Improvement’s Global Trigger Tool (GTT). The GTT is used to perform a retrospective review of a random sample of charts to 1) identify rates of AEs over time and 2) understand if system-level changes are improving hospital care.⁷ While the GTT has been used widely in general medical and surgical patient populations, it has less commonly been used in oncology populations.

The purpose of this study was to compare the frequency and type of AEs detected by a modified GTT as compared to a VSR system for identifying AEs in oncology patients who were enrolled in a larger clinical trial.

METHODS

This study was part of a prospective intervention study of a multi-faceted intervention to decrease AEs in hospitalized patients in intensive care units (ICUs) and oncology units.⁸ It was conducted across six oncology units at one academic hospital in Boston, Massachusetts from July 1, 2013 through May 29, 2015. The trial included 2,105 total patient admissions (1,030 before and 1,075 during the intervention). The clinical trial protocol was approved by the Partners HealthCare Institutional Review Board.

Study Unit Descriptions and Patient Eligibility

Each of the six oncology units had their own nurses, and physician teams traveled to each unit; physician teams included an attending physician, physician assistants, interns, and residents. Any patient 18 years or older and admitted to the one of the six oncology units in our study for 24 hours or longer was eligible to participate in the clinical trial. Each month during the study period, four clinical trial patients were selected randomly for a retrospective chart review with the GTT. The number of patients chosen was limited by the resources in the budget to hire registered nurses (RNs) for chart review.

GTT: Modification and Review Method for Study

This study used the IHI’s GTT from 2009, but eliminated the Surgical, Perinatal, and Emergency Department Module Triggers.⁷ Only two triggers from the Intensive Care Module Triggers were retained (See Supplementary Appendix 1).

The modified GTT was accessed through a web-based survey tool (Research Electronic Data Capture or REDCap tools⁹). At the beginning of the study, a sample of 13 charts were reviewed by the two nurses and the two physicians to improve review method consistency. Each patient’s medical record was reviewed retrospectively by one RN, who summarized the case, identified adverse events, and made an initial judgement about the severity and preventability of each event. We chose to hire RNs for this task because the patients in both the oncology and ICU populations had extremely complex courses. The GTT adapted the scale for severity from the National Coordinating Council for Medication Error Reporting and Prevention Index for Categorizing Errors.¹⁰ Other studies often only capture events with severity ratings E to I (indicating harm reached the patient¹⁰), we chose to utilize the full range of severity. Non-preventability was defined to align with previous work on adverse drug events.¹¹ For each case, the summary, description of AEs, and severity and preventability ratings were reviewed by two physicians. For patients where the physicians disagreed, a third physician reviewed the case. At the end of the study, AEs classified as “Other” were recategorized as “C15 Other,” “M13 Other medication,” or another more specific trigger by one of the physicians.

VSR: Patient Safety Policy, Routine Review, and AE Categories Included in Study

The GTT was compared to the hospital’s routine VSR method. Based on hospital patient safety policy, health care providers are instructed to report AEs immediately in the VSR system and provide the following details: when and where the event occurred, details about the person affected, specific details relevant to the event type (such as injury level), parties notified, and attachments. Medical directors, nursing directors, and practice managers are responsible for follow up. Within the Department of Quality and Safety, the Risk Management team includes four risk managers who review AE reports that are submitted via the VSR system daily. Depending on the AE, the report is either closed or kept open for a collaborative case review. If the AE is severe, the team conducts a root cause analysis that is shared with process owners for each AE type. The following AE types were obtained for the patients in this study because they were relevant to the domains of the modified GTT: falls, skin/tissue, medication/intravenous (IV) infusion, blood product/transfusion, and line/tube/drain. The VSR system was queried by the Department of Quality and Safety of the hospital. The research team conducted chart reviews for skin/tissue AEs to ensure they were not present on admission.

Comparison of AEs detected by both methods

The research team classified VSR AEs based on harm codes A-I for comparison to GTT AEs. After the conclusion of the clinical trial, a manual comparison of AEs detected by the GTT and VSR methods was conducted to determine which AEs were detected by both methods.

RESULTS

The random selection of four patients per month from the clinical trial resulted in a sample of 88 patients for this comparative study (Table 1). The average age of patients was 59 years. About two-thirds were male, two-thirds had private insurance, and the majority were white (85%). The average Charlson comorbidity score was 3.95 (SD 3.14) and average hospital length of stay was 19.53 days (SD 12.6).

Table 1.

Characteristics of Patients at the Time of Enrollment

Characteristics	N (%)
No. unique admissions	88
Race
White	75 (85.23)
Black	6 (6.82)
Hispanic	0 (0.00)
Other	4 (4.55)
Unknown	3 (3.41)
Insurance
Medicaid	2 (2.27)
Medicare	24 (27.27)
Private	57 (64.77)
Self-pay	0 (0.00)
Other	2 (2.27)
Missing	2 (2.27)
Sex
Female	32 (36.36)
Male	56 (63.63)
Age, mean (SD)	59.37 (13.57)
Charlson Comorbidity Index, mean (SD)	3.95 (3.14)
Median income based on zip code, mean (SD)	29589.56 (24319.62)
Total hospital length of stay per patient admission, mean (SD)	19.53 (12.60)

The modified GTT detected an AE rate of 0.90 AEs per patient (79 AEs in 88 patients) [95% (0.71, 1.12) AEs per patient]; 48 patients experienced an AE detected by the GTT while 40 patients did not. There was a predominance of AEs within the Medication Module (53/79; 67%), especially abrupt medication stops (16/52; 31%). Within the Care Module, AEs related to transfusion or use of blood products were the most common (8/25; 32%). Over half of AEs caused harm to patients (harm codes E to I; 41/79; 52%), but only about one-quarter were deemed preventable (21/79; 27%; Table 2).

Table 2.

Characteristics of Modified Global Trigger Tool Adverse Events

Event Type	Reported	Subgroup	N (total n=79)
Care Module (n=25)	C1	Transfusion or use of blood products	8
	C2	Code/arrest/rapid response team	2
	C4	Positive Blood Culture	1
	C7	Patient fall	2
	C9	Readmission Within 30 Days	1
	C11	Healthcare-associated infection	4
	C14	Any procedure complication	4
	C15	Other	3
Medication Module (n=52)	M4	Glucose less than 50 mg/dl	1
	M5	Rising BUN or serum creatinine	1
	M6	Vitamin K administration	1
	M7	Benadryl (Diphenhydramine) use	2
	M10	Anti-emetic use	5
		*zofran, ativan, compazine, zyprexia, reglan, vanco, omnipaque, gadovist
	M11	Over-sedation/hypotension	6
	M12	Abrupt Medication Stop	16
		*morphine, oxycodone, pepcid, amlodipine, lisinopril, lovenox, ASA, metoprolol, atenolol, oxycontin, dilaudid, haldol, dexamethasone, cefepime
	M13	Other	20
		*neupogen, allupurinol, oxycodin, dexamethasone, amodipine, hydralazine
Intensive Care Module (n=2)	I1	Pneumonia onset	1
	I3	In-Unit Procedure	1
Harm Code	Definition		N (total n=79)
A	Circumstances or events that have the capacity to cause error		1
B	An error that did not reach the patient		0
C	An error that reached the patient but did not cause harm		5
D	An error that reached the patient and required monitoring or intervention to confirm that it resulted in no harm to the patient		32
E	Temporary harm to the patient and required intervention		35
F	Temporary harm to the patient and required initial or prolonged hospitalization		5
G	Permanent patient harm		0
H	Intervention required to sustain life		0
I	Patient death		1
Preventability			N (total n=79)
Definitely Preventable			9
Probably Preventable			12
Probably Not Preventable			40
Definitely Not Preventable			18

^*medications taken directly from EHR description of Global Trigger Tool AE analysis during chart review

The VSR method detected an AE rate of 0.24 AEs per patient (21 AEs in 88 patients) [95% (0.15, 0.37) AEs per patient]; 14 patients experienced an AE detected by VSR while 74 did not. There was a predominance of AEs within the following event types: medication/IV (8/21; 38%), falls (6/21; 29%), and blood product/transfusion (4/21; 19%; Table 3). Over half of the AEs (70%) did not cause harm to patients.

Table 3.

Characteristics of Voluntary Reporting System Adverse Events (Type and Harm Codes)

Event Type		N (total n=21)
Falls		6
Skin/Tissue		1
Medication/IV		8
*ifosfamide, methotrexate, potassium/potassium chloride, busulfan, carboplatin, etoposide/etoposdie phosphate, mesna, dextrose 5%, 0.45% NaCl, Voriconazole, fenofibrate/fenofirbrate micronized, famotidine, prednisone
Blood Product/Transfusion		4
Line/Tube/Drain		2
Harm Code	Definition	N (total n=21)
A	Circumstances or events that have the capacity to cause error	4
B	An error that did not reach the patient	1
C	An error that reached the patient but did not cause harm	8
D	An error that reached the patient and required monitoring or intervention to confirm that it resulted in no harm to the patient	2
E	Temporary harm to the patient and required intervention	6
F	Temporary harm to the patient and required initial or prolonged hospitalization	0
G	Permanent patient harm	0
H	Intervention required to sustain life	0
I	Patient death	0

^*medications listed in event descriptions submitted to the Voluntary Safety Reporting System

A direct comparison of AE categories shows a different pattern between the two methods (Figure 1). VSR captured more falls but less skin/tissue events than the GTT. The GTT detected two falls (one unique and one that the VSR detected as well), while the VSR captured 4 additional falls, all of which were minor and did not result in injury.

Figure 1: Adverse Events Identified by Global Trigger Tool (n=79) versus Voluntary Safety Reporting (n=21).

AEs identified by the GTT (blue) are mapped onto VSR adverse events types (purple). There were six GTT AE types that did not align with VSR types, which are listed in the “other” column (green).

In contrast, the GTT captured more medication AEs (n=52), while VSR only captured eight events of this type. There was no overlap in medication AEs detected by both methods. This disparity of detection was also apparent for blood product AEs, where eight AEs were detected by GTT and three distinct AEs were captured by VSR.

With respect to harm codes, a majority of AEs captured by the modified GTT were classified as harm code E (temporary harm to the patient and required intervention; n=35; Table 2). A majority of AEs captured by VSR were classified as harm code C (an error that reached the patient but did not cause harm; n=8; Table 3). The GTT identified AEs which were harm codes F (temporary harm to the patient and required initial or prolonged hospitalization; n=5) and I (patient death; n=1), while the VSR did not capture any AEs with these harm codes.

Only two AEs (2/100; 2%) were identified using both methods (described by VSR event types): one fall and one line/tube/drain event (Figure 2). The one overlapping line/tube/drain AE was categorized as “any procedure complication” in the GTT. When comparing the descriptions for VSR versus GTT AEs, the VSR descriptions were more detailed and included a resolution comment, detailing how the AE was addressed (e.g. patient education, close monitoring, etc.). Of the total 98 unique AEs that emerged using both methods, majority were in the “medication” category.

Figure 2: Venn Diagram of Adverse Events Identified by Global Trigger Tool vs Voluntary Safety Reporting.

Of the 79 GTT AEs and 21 VSR AEs detected, only 2 AEs overlapped.

DISCUSSION

Only two AEs were identified by both methods. Also, the GTT detected AEs with higher harm codes than the VSR. We found that voluntary reporting was less likely to identify adverse drug events and events related to blood products than a trigger tool approach. On the other hand, voluntary reporting identified more falls than the trigger tool. The GTT did not detect as many falls as voluntary reporting because falls are not documented using a standardized process in the electronic health record.¹² However, the GTT identified more medication-related AEs.

More severe events are more often captured by GTT because these events have a more noticeable effect on the patient and lead to clinical actions, which is what the GTT is designed to capture.^{13 14} In addition, reasons for VSR under-reporting include hospital safety culture, poor guidelines surrounding VSR reporting, or a lack of follow up from leadership regarding the reported AE, thus decreasing the sense of self-efficacy for clinicians.^{15 16}

The reason that the GTT identified more medication-related AEs is because it includes many medication-related triggers that are easy to identify through the computerized provider order entry system and the electronic medication administration record. These systems only came into use over the past several decades, in part due to several studies demonstrating their benefit.^17–25 The GTT also performs well for blood product events due to the extensive electronic “paper” trail for blood products. These areas are promising for automated trigger detection methods. One major consideration is that many of the medications included in the GTT Medication Module are prescribed routinely in this population (such as anti-emetic medications) and that blood products are also used routinely. Therefore, efforts to modify the GTT modules are necessary. In contrast, VSR identified more falls. It is possible that the VSR system is more heavily used by nurses for falls because nurses are trained to immediately document these types of AEs in the VSR as opposed to the EHR.²⁶

Several groups have tried to improve the GTT for use in the oncology population. For example, Mattsson’s group tried to include an extra section for oncology but this did not improve GTT performance.^{27 28} Lipitz-Snyderman’s group is taking steps to create a completely new oncology trigger tool based on a literature review and modified Delphi process for use in inpatient and outpatient settings^{29 30}. Their team felt there was a need for a tool separate from the GTT for oncology patients, which could capture condition-specific AEs, AEs relevant to surgery (anesthesia or abscess related) and measurement error.³⁰ The creation of such a tool would exclude AEs resulting from the cancer disease process and focus on the quality of care delivered in order to facilitate timely identification of AEs and promote clinician intervention.

Study Limitations

This study was conducted at one academic hospital and the included patient population was primarily white, male, and privately insured. The fact that the chart review for GTT was capped at thirty minutes may have affected the results. In particular, the low prevalence of falls could be related because the risk of events like falls increases over longer hospital stays. Since the nurses/physicians started reviewing the chart at admission, later events could have been missed. However, the thirty minute time limit is part of the IHI GTT protocol.⁷ Another potential limitation is that only five AE types were obtained from the VSR system, thus AEs that were miscategorized could have been missed; this could have led us to underestimate the number of AEs identified by VSR.

CONCLUSION

Further studies exploring methods such as automated AE detection from electronic health records using natural language processing (NLP) and leveraging patient-reported AEs are needed.

Oncology patients have an increased risk of harm compared to general medical or surgical patient populations. It is important to refine safety surveillance methods for these patients. Yet, there is a dearth of studies in this area. We have shown that applying a general medical and surgical VSR system to an oncology patient population misses many AEs. A modified GTT detected more AEs, with the caveat that the majority of AEs were deemed not preventable due to the circumstances of the patients’ treatment course.

Ideally, we will be able to develop more sophisticated AE detection methods. Current research is focused on NLP to develop algorithms to identify AEs, particularly medication AEs.^{31 32} The Automated Adverse Event Detection Collaborative has also made progress in this area using NLP, but primarily focuses on pediatric care settings.³³ However, since clinicians do not document certain event types in standard locations in the electronic health record, such as falls, NLP methods alone cannot solve the problem.¹² Another avenue for AE detection, which does not require double documentation by nurses, is patient self-reported AEs. In this clinical trial, 66% of patients recorded information using a bedside patient portal.³⁴ In subsequent studies, we have shown how patient portals encourage patients to report safety concerns and enhance communication with providers.^35–37