SAFEST: Use of a Rubric to Teach Safety Reporting to Pediatric House Officers

Abstract

Background:

Among the many modalities of error detection in academic pediatric hospitals, patient safety reporting is an important component, particularly for unexpected events. Residents recognize the importance of reporting but cite some barriers to doing so. A rubric was developed to guide resident reporting and streamline information gathering in patient safety reports. The rubric used the acronym SAFEST as a reminder to include 6 key elements:

1. Staff involved in the incident.

2. Actual event description.

3. Follow-up initiated.

4. Effect on patient.

5. Standard of care described.

6. To-do/suggestions for improvement.

Objectives:

This study was designed to determine if the addition of this educational rubric into a standard quality improvement curriculum improves the consistency of information documented in patient safety reports as a subset of a larger quality improvement project aimed at improving safety reporting.

Methods:

A team of faculty members analyzed individual resident error reports for adherence to the 6 tenets of the SAFEST mnemonic.

Results:

From April to October of 2014, 2015, and 2016, a convenience sample of 131, 110, and 132 reports, respectively, were extracted and analyzed. For the rates of reporting “staff involved” and “standard of care,” the differences over time were significant, both with P values < 0.001. After training, residents were 2.2 times more likely to report on the “staff involved” in the error and 1.8 times more likely to report the “standard of care.”

Discussion:

These results describe successful education on a rubric designed to improve the content of patient safety reports.

INTRODUCTION

Prioritizing patient safety is essential to improving health care delivery.¹ According to the Institute of Medicine report, “To Err is Human,” over 1 million adverse events occur in the United States annually.¹ Reporting patient safety events, or “incident reporting,” is an important tool in reducing adverse events and “near misses.” The Institute of Medicine and Joint Commission recognize the benefits of patient safety reports,² and the Accreditation Council for Graduate Medical Education requires residents to “participate in identifying system errors and implementing potential systems solutions” as part of the Systems-Based Practice core competency.³ Physicians, however, often cite barriers to reporting patient safety events.⁴ These include uncertainty about reporting, when and what to disclose, limited feedback afterward, and potential negative professional repercussions.^5–7

A patient safety needs assessment showed that most pediatric residents at our academic center recognized their role in error identification but were not trained to report them. This finding is consistent with prior studies.^8–10 Also, prior institutional focus groups revealed that poor understanding of report review and closure led to residents devaluing the reporting process.

We developed a targeted intervention to understand these issues using Lean management principles; Lean is a practical philosophy that provides a framework to decrease waste and improve step-wise quality in health care and is widely used at our institution and taught during our core quality improvement curriculum. We initiated an exercise to improve the report review process with the goal of increasing reporting by assembling an interprofessional group of report reviewers (faculty physicians, nurse managers, bedside nurses, pharmacists, and risk management experts) who gathered with residents to map the patient safety reporting process and identify opportunities for improvement. Based on Lean techniques,^11,12 the group developed a value stream map (VSM) of the process (Fig. 1) highlighting steps with the most waste and lowest quality. We mapped this process from incident through “report closure” from the resident’s perspective. Although multiple areas for improvement were identified, much of the group VSM discussion centered on weaknesses of individual reports that lead to increased process time and waste time. Thus, an additional goal was identified: to improve the overall quality of reports filed. The VSM underlined key components of a safety report, which could lead to efficient and equitable investigation of safety events. To assist reporters in the process of filing, we developed a rubric to increase efficiency in root cause analysis of future safety events. We incorporated this rubric into the preexisting, standardized resident training on quality improvement and patient safety.¹² Its acronym “SAFEST” reminds reporters to aim to include these 6 key elements of an effective incident report (Fig. 2).

Fig. 1.

VSM of patient safety report process. The process is initiated at the incident and follows each step as determined by the interprofessional team. Areas of unsolved issues are noted by the storm cloud while the Kaizen burst represents the ideas generated from the team during the mapping exercise.

Fig. 2.

Description of the SAFEST rubric followed by necessary elements for each item to be scored.

Resident rubric training included analyses of sample patient safety reports and submission of new reports for review. The objective of this study was to determine if rubric training improves the quality of information documented in incident reports.

METHODS

This study meets the criteria of the University of Michigan Institutional Review Board for exemption. We completed a retrospective review of a convenience sample of patient safety reports filed by pediatric house officers in our institutional reporting system (rlSolutions, Cambridge, Mass.) between April and October 2014 as compared with the same months in 2015 and 2016. Reports submitted by individuals who had not received training in the rubric (nontrained fellows or nonpediatric trainees) were excluded. During that time, 72 pediatric residents and 24 medicine-pediatrics trainees underwent modular online training. During 2014, trainees had no formal training in patient safety reporting. During 2015 and 2016, over half of trainees experienced formal training using the SAFEST rubric as part of a longitudinal quality improvement and patient safety curriculum.¹¹ The other half had, at a minimum, been exposed to the rubric through discussions at monthly quality improvement meetings. A team of expert faculty (QI faculty mentors) reviewed all eligible incident reports. Figure 2 describes the consensus definitions for the SAFEST rubric components. We analyzed each report for inclusion of the 6 rubric tenets; these scores were reviewed by multiple team members (2–6 per report) to achieve consensus. Given that some incidents did not encompass all 6 tenets, we defined a “high-quality” report as containing at least 4 rubric components, based on focus group discussions with local report reviewers. We analyzed descriptive data using Microsoft Excel 2010, and comparative statistics by Chi-square and Chi-square Mantel Hanzel tests using SAS 9.4 (SAS Institute Inc., Cary, N.C.).

RESULTS

Figure 3 depicts the total number of incident reports by residents from 2011 through 2016. Our convenience sample includes resident reports from April through October in 2014, 2015, and 2016. During this time, we analyzed 131, 110, and 132 reports, respectively, with 23, 22, 25 reports excluded each year. After training, residents were 2.2 times more likely to report the “staff involved” component, and 1.8 times more likely to report the “standard of care” component (Fig. 4). All other measures showed no statistically significant difference. Figure 5 shows the percentage of high-quality reports (≥ 4 of the 6 rubric components described) over time, with significant improvement after training. Overall, posttraining reports were 2 times more likely to be of high quality (21% before training versus 41% after training, P < 0.05).

Fig. 3.

Total number of incident reports initiated by pediatric residents per year.

Fig. 4.

Percentage of high-quality reports pre- and posttraining. Statistically significant difference was noted in “Staff Involved” and “Standard of Care Described” categories (P < 0.05).

Fig. 5.

Percentage of “high-quality” patient safety reports per month.

DISCUSSION

Patient safety reports are an important, but insufficiently defined QI tool within many hospitals. These results describe the use of an educational framework for standardizing patient safety reporting at 1 institution. This standardization aimed to increase the utility of the reports institutionally and thus an interprofessional group of report reviewers designed this intervention. Our interprofessional team aimed to increase the report utility and decrease waste. This educational intervention focused on residents, who are responsible for submitting increasing numbers of patient safety reports. These results support the educational intervention leading to better reporting, as defined by the rubric.

The SAFEST rubric promotes standardized reporting. Reports containing most of these 6 components will streamline the review process, contributing to a Lean system. Another advantage of this rubric is the inclusion of real time improvement suggestions from involved reporters, which should foster a safety-focused environment.

This study had several limitations. Only one-half of the residents received formal training in the rubric, whereas the other half underwent a less formal exposure, and no control group with no exposure to the curriculum was available. There was also opportunity for bias as the analysis was done retrospectively by physicians who were not blinded to the timing (pre- versus postintervention) of the reports.

Iterative improvements include interventions to address the low inclusion categories, continued formal training and quarterly rubric reinforcement for all pediatric residents, education to medical scribes, dissemination to other faculty and staff, and linking high-quality reports to subsequent incident investigations. This has begun within our institution at house officer quality and safety committee and pediatric joint (medical, surgical, nursing) practice committee presentation and recognition. Also, ongoing discussions regarding updating the incident reporting system continue locally, although teaching effective reporting in any system was an important educational goal, knowing that many trainees will use different systems through the course of their careers.

CONCLUSIONS

With training, residents were more likely to include essential elements of effective incident reporting as defined by an institutional group of patient safety experts and reviewers. Further research is necessary to determine if this rubric adequately represents important elements for patient safety reporting outside of our institution, among nonphysician trainees, as well as the broader question of whether improved reporting actually leads to improved safety.