Abstract
Purpose:
Although patient safety is a priority in oncology, few tools measure adverse events (AEs) beyond treatment-related toxicities. The study objective was to assemble a set of clinical triggers in the medical record and assess the extent to which triggered events identified AEs.
Methods:
We performed a retrospective cohort study to assess the performance of an oncology medical record screening tool at a comprehensive cancer center. The study cohort included 400 patients age 18 years or older diagnosed with breast (n = 128), colorectal (n = 136), or lung cancer (n = 136), observed as in- and outpatients for up to 1 year.
Results:
We identified 790 triggers, or 1.98 triggers per patient (range, zero to 18 triggers). Three hundred four unique AEs were identified from medical record reviews and existing AE databases. The overall positive predictive value (PPV) of the original tool was 0.40 for total AEs and 0.15 for preventable or mitigable AEs. Examples of high-performing triggers included return to the operating room or interventional radiology within 30 days of surgery (PPV, 0.88 and 0.38 for total and preventable or mitigable AEs, respectively) and elevated blood glucose (> 250 mg/dL; PPV, 0.47 and 0.40 for total and preventable or mitigable AEs, respectively). The final modified tool included 49 triggers, with an overall PPV of 0.48 for total AEs and 0.18 for preventable or mitigable AEs.
Conclusion:
A valid medical record screening tool for AEs in oncology could offer a powerful new method for measuring and improving cancer care quality. Future improvements could optimize the tool’s efficiency and create automated electronic triggers for use in real-time AE detection and mitigation algorithms.
INTRODUCTION
Although patient safety is a priority in oncology, few tools measure adverse events (AEs) beyond treatment-related toxicities. This information is important to quantify the burden of harm experienced by patients with cancer and to identify opportunities for harm prevention. AEs refer to unwarranted outcomes resulting from medical care rather than the patients’ underlying disease or condition.1,2 Examples of AEs in oncology include lymphedema, anastomotic leak, and sepsis.
Attempts to develop methods for measuring errors and AEs in oncology care have been disappointing to date, generally relying on unsuccessful adaptation of approaches developed for general medicine.3-5 Accordingly, we assembled a set of clinical triggers in the medical record relevant across inpatient and outpatient settings in oncology.6 Because patients with cancer often receive complex care and are vulnerable to experiencing complications, a better understanding of potentially preventable harm is warranted.
The objective of this study was to assess the performance of our oncology screening tool in practice. We examined the extent to which triggered events identified AEs and preventable or mitigable AEs. A valid medical record screening tool for AEs in oncology could offer a powerful new method for measuring and improving cancer care quality.
METHODS
We performed a retrospective cohort study to assess the performance of an oncology medical record screening tool at Memorial Sloan Kettering Cancer Center, a National Cancer Institute–designated comprehensive cancer center. Medical record screening tools have been developed for different settings and populations to guide the identification of AEs for measurement and possible improvement.7-10 The oncology tool that we developed as part of the Cancer Harm (CHARM) study included 76 distinct triggers, or readily identifiable clinical indicators of potential AEs.6,11,12
The study cohort included 400 patients age ≥ 18 years diagnosed with breast (n = 128), colorectal (n = 136), or lung cancer (n = 136). Cohort patients started their first cancer-directed treatment at Memorial Sloan Kettering Cancer Center between January 1 and December 31, 2012, and were observed as in- and outpatients for up to 1 year or until death, whichever came first. We used stratified random sampling. For breast cancer, we stratified patients by stage and chemotherapy use. For colorectal cancer, we stratified patients by stage and cancer type (colon or rectal cancer), and for lung cancer, we also stratified patients by stage and cancer type (non–small-cell lung cancer or small-cell lung cancer).
Five trained nurses completed the chart reviews for the 400 patients. During the training process, a member of the study team reviewed the protocols and logistics. Then, the nurses compared three to five charts with each other that they completed independently to ensure consistency of the process and to discuss their preferred approach.
We allotted 1 hour for each patient chart. The nurses noted if they found any triggers and whether they were associated with an AE according to our study definition. They documented any key details about the patient’s case. Then, a nurse presented each case to physician reviewers who made the final determination as to whether the case met the study definition of AE, severity of harm, likelihood of preventability, and likelihood of harm mitigation. Inter-rater agreement was calculated.
We also obtained AE data from existing local safety event-reporting databases (Surgical Secondary Events for surgical complications and RL6:RISQ [RL Solutions, Toronto, Ontario, Canada] system for front-line staff reports). Candidate AEs were reviewed and coded independently by two physicians according to severity, preventability, harm mitigability, and AE type. An event was judged to be preventable if the AE resulted from clinical care that was inconsistent with standard oncology practice or was a treatment-related complication that should have been anticipated. An event was judged mitigable if the severity or the duration of harm could have been lessened had clinicians acted promptly and appropriately.
We calculated the percent agreement between physician pairs with regard to whether the event was an AE (1.0), the level of harm to the patient (categories A to D compared with E to I; 0.8), the likelihood of preventability (definitely or probably preventable, definitely or probably not preventable, or unable to determine; 0.8), and the likelihood of harm mitigation if the event was deemed not preventable (definitely or probably mitigatable, definitely or probably not mitigatable, or unable to determine; 0.8).
We calculated the tool’s overall positive predictive value (PPV) for identifying AEs and potentially preventable or mitigable AEs and PPVs of the individual triggers. PPV was defined as the number of times the trigger led to the identification of an AE divided by the total number of times the trigger was identified. We also calculated the sensitivity of the tool, using the combined, confirmed set of AEs from a review of both medical records plus the local reporting databases as the gold standard. On the basis of the PPV results and the research team’s expertise and experience, the investigators used a consensus-driven and iterative process to eliminate low-yield triggers and to produce a final set of oncology chart review triggers. Analyses were performed using Microsoft Excel (Microsoft, Redmond, WA) and SAS Software (SAS Institute, Cary, NC). This study was considered exempt research by the Institutional Review Board of Memorial Sloan Kettering Cancer Center.
RESULTS
Of the 400 patients, 32% were male, 19% were nonwhite, and 6% were Hispanic or Latino. Patients’ median age was 61 years (range, 26 to 92 years). We identified 790 triggers, or 1.98 triggers per patient (range, zero to 18 triggers). A total of 304 unique AEs were identified from all sources; 316 AEs, including repeat AEs, were identified from specific triggers. Examples of AEs included hypokalemia, mucositis, and Clostridium difficile. Thirty-six AEs were identified in the medical record without a specific trigger, and 14 were included in the hospital reporting systems alone. Thirty-three AEs were identified in both the hospital reporting system and the medical records. Thirty-six percent of patients (95% CI, 31% to 40%) had at least one AE.
The overall PPV of the original screening tool was 0.40 for total AEs and 0.15 for preventable or mitigable AEs. Examples of high-performing triggers for identifying AEs and preventable or mitigable AEs included return to the operating room or interventional radiology within 30 days of surgery (PPV, 0.88 and 0.38, respectively) and elevated blood glucose (> 250 mg/dL; PPV, 0.47 and 0.40, respectively). Poor-performing triggers included use of more than three doses of antiemetics within 24 hours (PPV, 0.07 and 0.02 for total AEs and preventable or mitigable AEs, respectively) and neurology consult and noncontrast head computed tomography (PPV, 0.06 and 0.00 for otal AEs and preventable or mitigable AEs, respectively; Table 1). The sensitivity of the medical record review was 92% compared with the gold standard.
Table 1.
Outcome of Triggers Sorted by Overall PPV
Of the 76 triggers in the tool, 18 were not identified through our study. The final modified screening tool included 49 triggers (Table 2). The PPV of the modified tool for identifying AEs was 0.48, and the PPV for identifying preventable or mitigable AEs was 0.18.
Table 2.
Final Memorial Sloan Kettering Cancer Center Oncology Adverse Event Trigger Tool
DISCUSSION
Our oncology-specific AE screening tool is the first effort, to our knowledge, to develop a medical record–based screening tool that is relevant across inpatient and outpatient oncology settings. We identified a large number of triggers and potential AEs in this longitudinal cohort of patients with cancer. Some of these AEs may be targets for prevention or harm reduction. This tool offers a more efficient approach than traditional chart review and may complement the toxicity-oriented tools in routine use (eg, the Common Toxicity Criteria).13
Our oncology tool’s performance is in line with that of tools in other clinical settings.7,14,15 It is important to note that it can be difficult to distinguish expected toxicities from unnecessary harm in oncology, which could influence our assessment of the tool’s performance for identifying preventable or mitigable AEs. We used best practice and required consensus by two physician reviewers but recognize the inherent subjectivity. The setting of this study was a single academic institution with a broad referral population and extensive clinical trials program. The institution’s focus on patient safety and quality may increase detection of AEs or reduce their incidence compared with other institutions. Further testing of the modified tool is required to generalize our results to other cancer care settings.
Future improvements could optimize the tool’s efficiency by creating automated electronic triggers for use in real-time AE detection and mitigation algorithms. It can also lead to more structured AE reporting toward this goal. Our oncology tool offers enhanced AE measurement in oncology, a step toward improving patient outcomes.
ACKNOWLEDGMENT
Supported by the United Hospital Fund and, in part, by the Cancer Center Support Grant to Memorial Sloan Kettering Cancer Center (Grant No. P30 CA 008748). Presented, in part, as a poster presentation at the 2016 AcademyHealth Annual Research Meeting, Boston, MA, June 26-28, 2016.
AUTHOR CONTRIBUTIONS
Conception and design: Allison Lipitz-Snyderman, David Classen, David Pfister, Aileen Killen, Andrew S. Epstein, Saul N. Weingart
Collection and assembly of data: Allison Lipitz-Snyderman, David Classen, David Pfister, Aileen Killen, Elizabeth Fortier, Andrew S. Epstein, Christopher Anderson, Saul N. Weingart
Data analysis and interpretation: Allison Lipitz-Snyderman, David Classen, David Pfister, Coral L. Atoria, Andrew S. Epstein, Saul N. Weingart
Manuscript writing: All authors
Final approval of manuscript: All authors
Accountable for all aspects of the work: All authors
AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
Performance of a Trigger Tool for Identifying Adverse Events in Oncology
The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/journal/jop/site/misc/ifc.xhtml.
Allison Lipitz-Snyderman
No relationship to disclose
David Classen
Employment: Pascal Metrics
Stock or Other Ownership: Pascal Metrics
Consulting or Advisory Role: Mentice, Phillips, Health Catalyst
Travel, Accommodations, Expenses: Mentice, Phillips, Health Catalyst
David Pfister
Consulting or Advisory Role: Boehringer Ingelheim
Research Funding: Boehringer Ingelheim, AstraZeneca, Exelixis, Genentech, Novartis, Eli Lilly, GlaxoSmithKline, Bayer, MedImmune, Merck
Aileen Killen
Employment: AIG
Coral L. Atoria
No relationship to disclose
Elizabeth Fortier
No relationship to disclose
Andrew S. Epstein
Other Relationship: UpToDate
Christopher Anderson
No relationship to disclose
Saul N. Weingart
Honoraria: UpToDate
REFERENCES
- 1.Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients: Results of the Harvard Medical Practice Study I. N Engl J Med. 1991;324:370–376. doi: 10.1056/NEJM199102073240604. [DOI] [PubMed] [Google Scholar]
- 2.Runciman W, Hibbert P, Thomson R, et al. Towards an international classification for patient safety: Key concepts and terms. Int J Qual Health Care. 2009;21:18–26. doi: 10.1093/intqhc/mzn057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mattsson TO, Knudsen JL, Brixen K, et al. Does adding an appended oncology module to the Global Trigger Tool increase its value? Int J Qual Health Care. 2014;26:553–560. doi: 10.1093/intqhc/mzu072. [DOI] [PubMed] [Google Scholar]
- 4.Mattsson TO, Knudsen JL, Lauritsen J, et al. Assessment of the global trigger tool to measure, monitor and evaluate patient safety in cancer patients: Reliability concerns are raised. BMJ Qual Saf. 2013;22:571–579. doi: 10.1136/bmjqs-2012-001219. [DOI] [PubMed] [Google Scholar]
- 5.Institute of Medicine . Delivering High-Quality Cancer Care: Charting a New Course for a System in Crisis. Washington, DC: The National Academies Press; 2013. [PubMed] [Google Scholar]
- 6.Lipitz-Snyderman A, Weingart SN, Anderson C, et al. ReCAP: Detection of potentially avoidable harm in oncology from patient medical records. J Oncol Pract. 2016;12:178–179, e224-e230. doi: 10.1200/JOP.2015.006874. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Sharek PJ, Horbar JD, Mason W, et al. Adverse events in the neonatal intensive care unit: Development, testing, and findings of an NICU-focused trigger tool to identify harm in North American NICUs. Pediatrics. 2006;118:1332–1340. doi: 10.1542/peds.2006-0565. [DOI] [PubMed] [Google Scholar]
- 8.de Wet C, Bowie P. The preliminary development and testing of a global trigger tool to detect error and patient harm in primary-care records. Postgrad Med J. 2009;85:176–180. doi: 10.1136/pgmj.2008.075788. [DOI] [PubMed] [Google Scholar]
- 9.Resar RK, Rozich JD, Simmonds T, et al. A trigger tool to identify adverse events in the intensive care unit. Jt Comm J Qual Patient Saf. 2006;32:585–590. doi: 10.1016/s1553-7250(06)32076-4. [DOI] [PubMed] [Google Scholar]
- 10.Unbeck M, Lindemalm S, Nydert P, et al. Validation of triggers and development of a pediatric trigger tool to identify adverse events. BMC Health Serv Res. 2014;14:655. doi: 10.1186/s12913-014-0655-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Classen DC, Lloyd RC, Provost L, et al. Development and evaluation of the Institute for Healthcare Improvement Global Trigger Tool. J Patient Saf. 2008;4:169–177. [Google Scholar]
- 12. Griffin F, Resar R: IHI Global Trigger Tool for Measuring Adverse Events (ed 2). Cambridge, MA, Institute for Healthcare Improvement, 2009.
- 13.Trotti A, Colevas AD, Setser A, et al. CTCAE v3.0: Development of a comprehensive grading system for the adverse effects of cancer treatment. Semin Radiat Oncol. 2003;13:176–181. doi: 10.1016/S1053-4296(03)00031-6. [DOI] [PubMed] [Google Scholar]
- 14.Matlow AG, Cronin CM, Flintoft V, et al. Description of the development and validation of the Canadian Paediatric Trigger Tool. BMJ Qual Saf. 2011;20:416–423. doi: 10.1136/bmjqs.2010.041152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Unbeck M, Schildmeijer K, Henriksson P, et al. Is detection of adverse events affected by record review methodology? An evaluation of the “Harvard Medical Practice Study” method and the “Global Trigger Tool.”. Patient Saf Surg. 2013;7:10. doi: 10.1186/1754-9493-7-10. [DOI] [PMC free article] [PubMed] [Google Scholar]