Impact of an Event Reporting System on Resident Complication Reporting in Plastic Surgery Training: Addressing an ACGME and Plastic Surgery Milestone Project Core Competency

STRUCTURED ABSTRACT

Background

The ACGME and Plastic Surgery Milestone Project have identified practice-based learning and improvement (PBLI), which involves systematically analyzing current practices and implementing changes for quality improvement, as a core competency in residency education. In surgical care, complication reporting is an essential component of PBLI as complications are analyzed in morbidity and mortality (M&M) conference for quality improvement. Unfortunately, current methods for capturing a comprehensive profile of complications for analysis at M&M conference are often inadequate and may significantly underestimate the true occurrence of complications. Therefore, the objectives of this study are to evaluate an intervention for complication reporting, and compare this to current practice, in a plastic surgery training program.

Methods

This is a pre- and post-intervention study evaluating resident reporting of complications on a plastic surgery service. The intervention was an online event reporting system developed by department leadership and patient safety experts. The pre- and post-cohorts consisted of all patients having surgery during two separate 3-month blocks bridged by an implementation period. An additional cohort was evaluated 7-months after the first post-intervention group, over a 1-month block, to analyze consistency over time. A trained reviewer recorded complications and this served as the reference standard. Fisher’s exact test was used for binary comparisons.

Results

There were 32 complications detected in 219 patients from June–August, 2015, 35 complications in 202 patients from October–December, 2015, and 9 complications in 91 patients from August–September, 2016. The proportion of complications reported for M&M conference in the pre-intervention group was 28.1% (9/32). After the intervention, this significantly increased to 91.4% (32/35) (p < 0.001). When allowing for a time lapse, the proportion reported (88%, 8/9) remained increased (p = 0.002).

Conclusions

An intervention utilizing an event reporting system, supported by departmental leadership, led to significant improvements in complication reporting by plastic surgery residents.

INTRODUCTION

The imperative for quality health care is integral to the foundation of modern medicine. In 1999, the Institute of Medicine (IOM) released the landmark report, To Err is Human: Building a Safer Health System, and concluded that health care in the United States is not as safe as it should be due to high numbers of adverse events resulting from preventable medical errors.¹ Beyond the morbidity and mortality caused by medical error, the overall cost to society is approximately $400 billion to $1 trillion, an astronomical amount representing nearly 50% of our nation’s healthcare expenditures.² Medical error is intricately tied to surgical care. At least 50%, and up to 66%, of adverse events in hospitals are related to surgical operations and more than half of these result from preventable errors.^3–6 Despite the release of the IOM report nearly two decades ago, progress in reducing the rates of harm and adverse events has remained stagnant.⁷ In an effort to improve patient care and systematically improve the health care system, the Centers for Medicare and Medicaid Services (CMS), the Joint Commission, and the Agency for Healthcare Research and Quality (AHRQ) have emphasized the need for healthcare institutions and providers to prioritize clinical quality improvement initiatives.

Recognizing the foundational role of quality improvement in the education of physicians, the Accreditation Council for Graduate Medical Education (ACGME) now mandates all trainees in graduate medical education (GME) programs achieve competency in Practice-Based Learning and Improvement (PBLI) and Systems-Based Practice (SBP) prior to residency graduation.^8,9 Simply considered, these competencies require trainees to “systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvements” and to navigate the complex healthcare system and participate in continually improving it.^8,10 Traditionally, surgical residency programs have utilized the morbidity and mortality (M&M) conference as the primary method for analyzing current practices and complications in an effort to improve quality. While M&M conferences have a valuable role in surgical education, these conferences are only beneficial if they accurately reflect the complications occurring over the defined time period.^11,12 Unfortunately, current methods for capturing a comprehensive profile of complications for analysis at M&M conference are often inadequate.^13–15 The most common of these methods include the chief resident on service reporting all complications at set time points or all residents on service maintaining their own list of complications, which are subsequently collated prior to M&M conference.^11,13 Prior studies have found these traditional methods significantly underestimate the true occurrence of complications, with more than 75% of complications not reported.^13,14 This presents a dilemma for resident education in quality improvement as it is widely recognized that an integral component of quality improvement is the use of valid and transparent measures to collect and report performance and outcomes.¹⁶

In Plastic Surgery, accurate reporting and evaluation of complications is critical for quality improvement and resident education, and a core component of the Plastic Surgery Milestone Project.^17–19 Not only does it afford the opportunity to learn from experience, but also enables a comparison of treatment strategies and techniques to facilitate the practice of evidence-based medicine.²⁰ The objective of this study was to develop, implement, and evaluate an intervention for complication reporting, and compare this to current practice, in a plastic surgery training program.

METHODS

Study Overview

This is a pre- and post-intervention study evaluating resident reporting of complications and adverse patient events on a plastic surgery service in a teaching hospital from June 2015 to September 2016. The pre-intervention and post-intervention cohorts consisted of all patients having surgery on the pediatric plastic surgery service during two separate 3-month blocks bridged by a transition period for intervention implementation. The pre-intervention evaluation occurred from June 2015 to August 2015 and the post-intervention evaluation occurred from October 2015 to December 2015. Additionally, a further cohort was evaluated beginning 7-months after the end of the first post-intervention study arm, over a 1-month block from August 1^st, 2016 to September 1^st, 2016, to analyze consistency over time.

Intervention Development and Implementation

This quality improvement initiative followed the overall design of the Plan-Do-Study-Act (PDSA) Cycle (Figure 1).²¹ The PDSA method is widely used in healthcare quality improvement initiatives.²² Leadership from the Department of Surgery, the Division of Plastic and Reconstructive Surgery, and the Patient Safety Office at Washington University School of Medicine in St. Louis convened to review best practices in patient safety and quality improvement. We formed a Patient Safety Work Group with resident, faculty, and staff members and placed an emphasis on improving practice patterns related to adverse event and complication reporting, among other focus areas. Faculty representation comprised of at least two faculty members (senior-level and junior-level) from each Division in the Department of Surgery and resident representation comprised of both junior and senior residents in all surgical training programs in the Department of Surgery. Representatives of the Patient Safety Office met regularly with members of this work group to establish an overall online platform for adverse event reporting and integrated feedback from these meetings to optimize efficiency and ease of use. Furthermore, each Division and Section had specific input in the development of the event reporting system (ERS) to create a customized user interface that reflects the unique adverse events and complications relevant to that particular specialty.

Figure 1.

Overview of study design, following the Plan-Do-Study-Act model, for this quality improvement project.

We defined complications as “any deviation from the normal postoperative course” in accordance with widely accepted terminology originating from the Clavien-Strasberg (T92) system, subsequently modified into the Clavien-Dindo system and the Accordion System, for classifying surgical complications.^23–25 We defined an adverse event as “an event that results in unintended harm to the patient by an act of commission or omission rather than by the underlying disease or condition of the patient” according to the National Quality Forum and IOM terminology.^26,27 Complications and adverse events were grouped together and considered at the patient-level. For the pediatric plastic surgery service, the authors reviewed the literature and institutional experience to identify all procedures performed and all potential complications and/or adverse events.

The ERS was integrated as a desktop icon into all computers in the emergency department, surgery clinics, operating rooms, and offices to facilitate use. To minimize reporting burden, the time to submit an event was repeatedly tested and the ERS modified until a report could be documented in less than one minute. After clicking on the desktop icon, a “quick submit” icon appears and is followed by a brief submission form that includes drop down lists and pre-populated items based on the nature of the event reported (Figure 2) (See Figure, Supplemental Digital Content 1, which shows an online event reporting system user interface for the “Quick Submission” report form demonstrating items required to complete report, drop-down menu items, and potential for write-in descriptions when necessary, INSERT LINK). Summary reports of all complications reported are sent weekly to leadership in the Division of Plastic and Reconstructive Surgery and a monthly report is formulated for use at M&M conference. Furthermore, all residents were provided with a written report summarizing complications prior to M&M and each complication was discussed during M&M conference, with feedback provided by faculty members.

Figure 2.

Online event reporting system user interface demonstrating feasibility of use and “Quick Submit” option for efficiency.

Data Collection and Analysis

A trained surgical reviewer recorded all complications and adverse events for consecutive patients undergoing operations on the pediatric plastic surgery service throughout the study period. These data served as the reference standard. The surgical reviewer was a clinical outcomes and patient safety fellow at our institution and underwent training on the essentials of data collection, reporting, database management, computer knowledge, hospital systems knowledge, and hospital departmental knowledge prior to starting. Complications were included if they occurred within 90 days of the procedure. Data were collected in two separate 3-month blocks and a final 1-month block, as referenced above. In the pre-intervention group, resident reporting of complications was consistent with current practices at that time at our institution where the chief resident on service reports complications, after discussions with team members, prior to monthly M&M conferences. The post-intervention groups included all residents on service utilizing the ERS for complication reporting at the time of the event and the ERS was synchronized to generate data for M&M conference. All residents in the Department of Surgery were informed of the need for improved M&M reporting prior to the start of this study (before the pre-intervention group). The department’s initiatives to improve patient safety and quality improvement were communicated to all residents prior to the start of the academic year and residents were continually updated on progress via teaching conferences and monthly emails.

Statistical analysis was performed using established methods, with Fisher’s exact test used for binary comparisons of the independent samples. Two-sided α = 0.05 with Bonferroni correction for multiple comparisons was set a priori to detect significance. All statistical analyses were conducted using commercially available software (SAS 9.4; SAS Institute, Cary, NC).

RESULTS

The pre-intervention study cohort comprised of 219 patients from June 2015 to August 2015. The first post-intervention study cohort comprised of 202 patients from October 2015 to December 2015 and the final post-intervention study cohort comprised of 91 patients in August 2016. There were 32 complications detected in the pre-intervention group, 35 complications detected in the first post-intervention group, and 9 complications detected in the final post-intervention group using the reference standard. There was no significant difference between the occurrence of complications in the pre-intervention group (32/219, 14.6%) when compared to the first post-intervention group (35/202, 17.3%, p = 0.466) or when compared to second post-intervention group (9/91, 9.9%, p = 0.316). The grouped distribution of complications and adverse events, by relative frequency, is demonstrated in Table 1. The distribution of complications based on severity grade, according to the Modified/Expanded Accordion complication classification system, is demonstrated in Table 2. Furthermore, the percentage of complications not reported, grouped by categories, is presented in Table 3 for each of the groups.

Table 1.

Occurrence of complications and adverse events, grouped into categories, for the pre-intervention and post-intervention patient cohorts.

Complications, by group	Pre-Intervention (N=219)	Post-Intervention Group 1 (N=202)	Post-Intervention Group 2 (N = 91)
Wound Complication	11 (5.0%)	12 (5.9%)	3 (3.3%)
Surgical Site Infection	7 (3.2%)	8 (4.0%)	1 (1.1%)
Unplanned Reoperation	4 (1.8%)	5 (2.5%)	2 (2.2%)
Unplanned Transfer to ICU	2 (0.9%)	2 (1.0%)	0 (0.0%)
Unplanned Readmission	1 (0.5%)	2 (1.0%)	1 (1.1%)
Medical Complication	4 (1.8%)	4 (2.0%)	1 (1.1%)
Intraoperative Adverse Event	3 (1.4%)	2 (1.0%)	1 (1.1%)
Mortality	0 (0.0%)	0 (0.0%)	0 (0.0%)
Total	32 (14.6%)	35 (17.3%)	9 (9.9%)

Table 2.

Severity of complications and adverse events categorized using the Expanded Accordion Severity Grading System for the pre-intervention and post-intervention cohorts.²⁵

Complication Severity, by Expanded Accordion System	Pre-Intervention (N = 32)	Post-Intervention Group 1 (N = 35)	Post-Intervention Group 2 (N = 9)
Mild	18 (56.2%)	18 (51.4%)	5 (55.6%)
Moderate	5 (15.6%)	6 (17.1%)	1 (11.1%)
Severe: invasive procedure not requiring general anaesthesia	4 (12.5%)	5 (14.3%)	1 (11.1%)
Severe: invasive procedure requiring general anesthesia	5 (15.6%)	6 (17.1%)	2 (22.2%)
Severe: organ system failure and invasive procedure under general anesthesia	0 (0.0%)	0 (0.0%)	0 (0.0%)
Death	0 (0.0%)	0 (0.0%)	0 (0.0%)

Table 3.

Complications not reported, grouped into categories, for the pre-intervention and post-intervention patient cohorts.

Complications Not Reported*, by group	Pre-Intervention (% Not Reported)	Post-Intervention Group 1 (% Not Reported)	Post-Intervention Group 2 (% Not Reported)
Wound Complication	81.2% (9/11)	All reported	All reported
Surgical Site Infection	57.1% (4/7)	All reported	All reported
Unplanned Reoperation	75.0% (3/4)	20.0% (1/5)	50.0% (1/2)
Unplanned Transfer to ICU	50.0% (1/2)	50.0% (1/2)	N/A
Unplanned Readmission	100.0% (1/1)	All reported	All reported
Medical Complication	75.0% (3/4)	All reported	All reported
Intraoperative Adverse Event	66.7% (2/3)	50.0% (1/2)	All reported
Mortality	N/A	N/A	N/A
Total	N=23	N=3	N=1

^*There were N=23 complications not reported out of N=32 total complications in the pre-intervention group, N=3 complications not reported out of N=35 total complications in post-intervention group 1, and N=1 complications not reported out of N=9 total complications in post-intervention group 2

Compared to the reference standard, the proportion of complications reported by residents for M&M conference in the pre-intervention group was 28.1% (9/32). After implementation of the ERS intervention, the proportion of complications reported by residents significantly improved to 91.4% (32/35) (p < 0.001) (Figure 3). When allowing for a time lapse of 7-months, the proportion of complications reported by residents (88%, 8/9) remained significantly increased compared to the pre-intervention group (p = 0.002) and statistically equivalent to the first post-intervention group (p = 0.783).

Figure 3.

Comparison of the proportion of complications reported by residents in the pre-intervention, first post-intervention, and second post-intervention study groups. *Graph demonstrates significant improvements in the proportion of complications reported after implementation of the intervention (9/32, or 28.1%, of complications reported pre-intervention to 32/35, or 91.4%, of complications reported post-intervention, p < 0.001). **This result remained stable over time with a significantly increased proportion of complications reported in the second post-intervention group (88%, 8/9) compared to the pre-intervention group (p = 0.002).

DISCUSSION

In this study, we found that a simple intervention utilizing an online event reporting system significantly improved complication reporting by plastic surgery residents in an academic training program. To date, few studies have evaluated the potential benefit of different interventions to improve complication reporting among surgical residents. This study expands the current literature and demonstrates the utility of event reporting systems in surgical training. Furthermore, this study demonstrates the value of leadership-initiated and supported quality improvement programs to meet ACGME core competency requirements in surgical training. Prior to utilization of the surgeon-designed ERS, residents only reported approximately 25% of complications. Following implementation and adoption of the ERS intervention, residents significantly improved to report approximately 90% of complications for review, creating the potential for substantial quality improvement over baseline practice. Furthermore, these improvements remained stable over time, indicating the intervention may have a durable effect.

The ability to accurately identify, report, and analyze complications is integral to quality improvement in surgical care and a core competency requirement for all plastic surgery residents in the United States.^8,18,20,28 A thorough review of errors, complications, and adverse events provides trainees the opportunity to learn from experience, identifies areas for targeted safety interventions, and promotes implementation of best practices for continued quality improvement.^29–31 Furthermore, the ability to accurately document and thoroughly discuss complications is a critical skill all residents must develop to be prepared for independent practice.^9,12,32,33 Prior studies have demonstrated patients are often unsatisfied with physicians’ disclosure of complications and medical errors, with lack of information, emotional support, and thoroughness of discussion being the most prevalent reasons cited.^34,35 Poor communication about complications can deteriorate the physician-patient relationship and is associated with a significantly greater likelihood of malpractice lawsuits when compared to open and thorough discussion.^33,36–38 Additionally, the AHRQ Hospital Survey on Patient Safety Culture demonstrated that voluntary reporting of adverse events is associated with improved perceptions of safety culture and practices.³⁹

While the low proportion of complications reported in the pre-intervention group in our study may be surprising, it underscores the limitations of traditional methods for capturing a comprehensive profile of complications. Prior studies in the surgical literature have documented that conventional reporting methods for M&M conferences led residents to substantially underreport inpatient and postdischarge complications.^13–15 Furthermore, a prior survey of surgical residents revealed less than 30% frequently reported complications on their service to M&M conference and over 40% had never reported a complication to M&M conference.¹⁵ This represents a potentially concerning gap in attaining competency in the practice-based learning and improvement ACGME core competency.⁸ Additionally, the Plastic Surgery Milestone Project has also identified complication reporting and analysis as a key component of patient safety milestones in the systems-based practice competency.¹⁸ Utilization of an ERS, such as the one proposed in this study, can substantially improve on current practices and promote competency in both the general ACGME requirements and the Plastic Surgery Milestone targets.

Event reporting systems (ERS), also known as incident reporting systems (IRS), are widely utilized in a multitude of high-risk industries, including the aviation and nuclear industry. Simply considered, ERS provide an efficient way to identify risks and adverse outcomes so interventions can be implemented to subsequently mitigate these risks and outcomes.⁴⁰ In healthcare, ERS have evolved to become a key component of improving patient safety after the Patient Safety and Quality Improvement Act of 2005 and the Joint Commission requirement for hospitals to have a system for adverse event reporting.^40–42 However, ERS are just one of many available methods for detection and reporting of adverse events, errors, and complications. Another potential option is the “Trigger Tool” methodology, which involves a retrospective review of a random sample of patient records using a list of “triggers” or cues to potential adverse events and complications in the medical record.⁴³ This methodology has been applied across healthcare and “triggers” can be personalized to the area of quality improvement focus or the specific discipline.⁴⁴ An additional method utilized in surgery is the American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP). In this program, a trained nurse reviewer collects 30-day complication data using standardized definitions for a sample of cases performed during an 8-day cycle each month and these data can be linked to a web-based system.⁴⁵

All of these methods have purported benefits and challenges. The ACS-NSQIP system, as it relates to complication reporting and review, has limited application in plastic surgery because it only samples a small proportion of plastic surgery operations and 30-day outcomes are often inadequate for many plastic surgery operations.^46–48 ERS also have challenges that affect implementation and use. By design, ERS are dependent on voluntary reporting and can be affected by miscommunication. In our study, we still did not achieve a 100% rate of complication reporting in the post-intervention groups. Our hypothesis for why this occurred is there were miscommunications between residents regarding who was going to enter the complication into the event reporting system. Anecdotally, we discussed with residents involved in cases that had complications not reported in the post-intervention group. All of these situations involved two, or more, residents who each thought the other would enter the complication in the ERS system. Therefore, we have now emphasized to all our residents a clear hierarchy of responsibility for reporting when more than one resident encounters a complication: it is the responsibility of the most senior resident to enter the complication in the ERS system. It is our opinion that this will prevent future miscommunications and further improve the reporting rate. An additional challenge to voluntary reporting is that complications reported in the ERS can be reviewed by the institution separately from the reporting physician. This requires a culture of safety to exist at the institutional level, leadership involvement in establishing a pattern of reporting, policy incentives to ensure reporting, and all team members to be actively engaged in the mission of quality improvement. Potential barriers to reporting include the burden of use, inadequate feedback following a report, and unclear expectations of which events/complications to report.⁴⁹ Therefore, for successful implementation and acceptance of an event reporting system, an effort must be made to simplify the process of reporting, clarify what needs to be reported, provide prompt and comprehensive feedback on reports, and establish a culture of safety with transparency and open discussion among team members.

Ultimately, ERS must be responsive to the local environment of intended use. Our ERS was pilot tested in the section of pediatric plastic surgery and we have now expanded utilization across our Division and Department given the favorable results in this study. We believe our experience was successful because we made reporting simple (less than 1 minute to complete a “quick submission” report), had leadership and team member buy-in, and discussed each report in person to ensure adequate feedback. Of note, despite an increase in the overall volume of complications reported, we did not experience significant challenges in regards to the time available for discussing each complication in M&M conference. At our institution, we have teaching conferences for 3 hours once weekly and M&M conference is allotted 1.5 hours of these 3 hours once monthly. As all faculty and residents are provided a list and summary of complications, generated from the ERS, for review prior to M&M, we are able to efficiently discuss complications in a range of 5–8 minutes. Of course, time is allowed for more in-depth discussions as needed. Certainly, programs that allot less time for M&M conference may encounter difficulties with comprehensively reviewing each complication if reporting substantially improves. This would then require structural changes in teaching conferences to accommodate these discussions and it would be important to get faculty and resident buy-in prior to making these changes. Following this template will allow other institutions to use a similar system to improve resident reporting in their local environment.

This study also highlights the value of plastic surgery resident engagement in quality improvement initiatives on a departmental and hospital level. The highest level, Level 5, of the ACGME Plastic Surgery Milestone Project target for systems-based practice involves leading or participating in a multidisciplinary team to address patient safety issues.¹⁸ Implementing improvement initiatives and programs in clinical medicine requires substantial participation and buy-in from all members of the patient care team. Residents are a critical component of that team; however, to date, residents have had limited engagement in quality improvement initiatives. Not surprisingly, the 2016 ACGME Clinical Learning Environment Review (CLER) program, designed to review the nearly 300 medical institutions and hospitals that support almost 9,000 GME training programs, reported substantial deficiencies in 2016 regarding resident knowledge of and engagement in quality improvement initiatives.⁵⁰ This is a limitation in surgical education that needs to be addressed. Resident-led and leadership supported quality improvement projects are integral to preparing trainees for independent practice in a complex healthcare system that is increasingly emphasizing value and safety in patient care.

This study has limitations that merit discussion. A randomized-controlled trial design was not feasible, and due to the nature of the intervention and goal of improving the safety culture at our institution, residents were aware of the overall aims of this quality improvement program. This introduces the possibility the results we observed are attributable to a Hawthorne effect. To mitigate this, we analyzed data over the period of 13-months and allowed a time lapse in between the first post-intervention group and the second post-intervention group. At the time of data collection for the second post-intervention group, all residents in our Department had been using the ERS for several months and were unaware their reporting patterns were still being analyzed. Given that both post-intervention groups had significant improvements compared to the baseline pre-intervention group, we believe these changes are durable and our results would be stable if evaluated in the future. Also, it is unclear if our results reflect a true, lasting cultural change or if they reflect the benefit of integrating technology into clinical practice. Qualitative interview or survey data would be the best methodology to measure a true cultural change and future studies evaluating plastic surgery residents’ attitudes to complication reporting and discussion would be valuable for continued quality improvement.

CONCLUSIONS

An intervention utilizing an online event reporting system led to significant improvements in complication reporting by plastic surgery residents in an academic teaching hospital. Implementation of an event reporting system can enhance practice-based learning and quality improvement, addressing both ACGME and Plastic Surgery Milestone Project competency targets for residents. Furthermore, resident involvement in quality improvement initiatives is essential to training physicians for independent practice in a complex healthcare system.

Supplementary Material

Supplemental Digital Content 1

Supplemental Digital Content 1. Figure shows an online event reporting system user interface for the “Quick Submission” report form demonstrating items required to complete report, drop-down menu items, and potential for write-in descriptions when necessary, INSERT LINK.