Patient Safety Checklists: Do They Improve Patient Safety for Supracondylar Humerus Fractures?

Abstract

BACKGROUND

The purpose of the study was to investigate whether a safety checklist could be used consistently in an academic center, and, whether its presence correlates with a decreased rate of complications, and therefore, improved overall patient safety.

METHODS

Data from 3 years before and after the implementation of the checklist were compared. Pre-checklist data from August 2008 through August of 2011, including all operative supracondylar humerus fractures treated at our institution, were retrospectively reviewed. Post-checklist data, from August 2011 to August 2014 were prospectively collected. Patients’ charts and their imaging were all reviewed for: fracture type, nerve injury, placement of a medial pin, infection, loss of alignment, loss of fixation, and return to the operating room. Patients who were within the checklist group were reviewed for checklist compliance and concordance of resident and attending-attested checklists.

RESULTS

931 operative supracondylar humerus fractures were reviewed - 394 in the pre-checklist group and 537 in the post-checklist group. There was no significant difference in fracture type between the pre- and post-checklist groups. No significant differences were found between pre- and post-checklist patients in regards to loss of fixation, loss of alignment, infection, or nerve injury. In the post-checklist group, the number of medial pins placed was significantly less than in the pre-checklist group [p = 0.0001], but this was not found to have clinical significance. In the pre-checklist group, 11 patients returned to the operating room for a second procedure whereas 4 in the post-checklist group had a return to the operating room. This finding was significant [p = 0.015], but the returns to the operating room were not related to checklist parameters. The checklist compliance of the attending physicians was 85.85% and the residents were compliant 83.11% of the time. There were documented discrepancies between resident and attending checklists in 7.38% of all total checklists.

CONCLUSIONS

Our patient safety checklists are not necessarily affecting patient care in a clinically significant manner. It is important that we validate and refine these specialty-specific checklists before becoming reliant on them.

LEVEL OF EVIDENCE

III

INTRODUCTION

As healthcare continues to evolve, evidence and documentation are becoming more integral to our treatment of patients. We are expected to standardize treatments, and base those off of validated recommendations and outcome measures. Safety checklists are a popular way to improve patient care and reduce errors. Janssen et al recently reported 11% of Orthopaedic Surgeons surveyed suggested that some manner of checklist protocol is an important means of improving patient safety, with only training (20%) and reporting (12%) being more frequently suggested¹. Despite their growing popularity, the efficacy of these checklists in standardizing surgical interventions and management for specific diagnoses in Orthopaedics is largely unproven.

Supracondylar humerus fractures account for 3% of all pediatric fractures^{3, 4} with an annual rate of 177.3 per 100,000 children⁵. The pinning of supracondylar humerus fractures is the most common surgical procedure performed by ABOS Pediatric Orthopaedic Diplomats⁶. With approximately 1300 publications regarding this fracture present in the literature, an available Clinical Practice Guideline (CPG), and the high incidence of this injury in the pediatric population, our institution determined that this diagnosis was appropriate for the creation of checklist protocol and clinical pathway.

The purpose of the study was to investigate whether a safety checklist could be consistently used in an academic center, and, whether its presence correlated with a decreased rate of complications, and therefore, improved overall patient safety.

MATERIALS AND METHODS

Patient safety checklists, based on AAOS Clinical Practice Guideline (CPG) for Supracondylar Humerus Fractures², were embedded within our electronic medical record (EMR) in 2011 for the pre-operative, post-operative, and discharge evaluations of all operative supracondylar humerus fractures. All attending staff, residents, and mid-level providers were made aware of the checklist system, and instructed regarding its usage and its goals. Regular reminders and updates were given at staff meetings.

The resident or mid-level provider is automatically prompted to complete each checklist throughout the patient’s admission via our EMR, and the attending staff is notified electronically when they are supposed to attest the completed resident checklist.

The pre-operative checklist includes documentation of appropriate imaging, position of immobilization, neurovascular exam and surgical requirement. The post-operative checklist includes documentation of pin configuration, whether or not an open reduction was required, vascular status, intraoperative imaging, and position of immobilization. The discharge checklist includes a neurovascular examination only (Appendix).

Any deviation from the standard set forth by the checklist, or any negative answer, prompts the user to give an explanation for the difference. For example, if a medial pin is placed, the surgeon must give a reason in a free text box prior to being able to complete the checklist.

After institutional review board approval was obtained, data from 3 years before and after the implementation of the checklist were compared. Data from from August 2008 through August of 2011, including all operative supracondylar humerus fractures treated at our institution, were retrospectively reviewed. This comprised the ‘Pre-Checklist’ group. Data from August 2011 to August 2014, for all patients within the checklist group regardless of checklist completion, were prospectively collected and retrospectively reviewed. This comprised the ‘Post-Checklist’ group. Patients’ charts and their imaging, in both the pre and post-checklist groups, were all reviewed for: fracture type, nerve injury, placement of a medial pin, infection, loss of alignment, loss of fixation, and return to the operating room. All fractures were classified according to the Wilkins Modification of the Gartland Classification.⁷ Loss of alignment was defined as a radiographic change in Baumann’s Angle of 10° or greater, and/or a change in the relationship of the anterior humeral line to the capitellum at the time of post-operative follow-up. Loss of fixation was defined as noted pin migration radiographically at the time of post-operative follow-up. Patients who were within the checklist group were reviewed for checklist compliance and concordance of resident and attending-attested checklists.

A power analysis pre-calculation showed that a decrease of complications from 5% to 1-2% would be detected with 95% power while maintaining a two-sided Type I error rate of α=0.05. Statistical significance was set at p ≤ 0.05 and all confidence intervals at 95%.

Baseline demographic characteristics and pre and post checklist differences were assessed using two-sample t-tests for continuous variables and Chi-square test for categorical variables. Specific groups (return to OR and medial pin placement) were further analyzed in order to determine whether the differences present were affected by other factors.

RESULTS

A total of 931 operative supracondylar humerus fractures were reviewed. Of them, 394 fractures were reviewed in the pre-checklist group and 537 in the post-checklist group. All operative cases were performed with resident and/or fellow assistance under the direct supervision of attending physicians. There was no significant difference in fracture type, and therefore, injury severity, between the pre- and post-checklist groups (Table 1). The number of pins used to fix the fractures was noted by fracture type both pre- and post-checklist (Tables 2A and 2B).

Table 1.

Fracture Classification Pre vs. Post Checklist

	Pre-Checklist	Post-Checklist
Type II	134	157
Type III	243	357
Flexion Type	12	22
Unstable in Flexion/Extension	1	5

Table 2A.

Number of Pins Used for Fixation Pre-Checklist

	2 PINS	3 PINS	4 PINS	5 PINS
FLEXION TYPE	5	7	0	0
TYPE 2	114	20	0	0
TYPE 3	106	130	5	2
UNSTABLE IN FLEXION/EXTENSION	0	1	0	0

Table 2B.

Number of Pins Used for Fixation Post-Checklist

	2 PINS	3 PINS	4 PINS	5 PINS
FLEXION TYPE	8	14	0	0
TYPE 2	118	39	0	0
TYPE 3	75	276	5	1
UNSTABLE IN FLEXION/EXTENSION	1	1	3	0

No significant differences were found between pre- and post-checklist patients in regards to loss of fixation, loss of alignment, infection, or nerve injury (Table 3A). The nerve injuries were further analyzed to further characterize the nerve injuries in each group and to determine if they were present pre-operatively, and therefore, presumably not iatrogenic in nature. The comparisons are shown in Table 3B.

Table 3A.

Complications Pre vs. Post Checklist

	Pre-Checklist	Post-Checklist	p-value
Loss of Fixation	5	1	0.08
Loss of Alignment	22	21	0.23
Infection	6	2	0.07
Nerve Injury	40*	44*	0.35

Table 3B.

Characterization of Nerve Injuries Pre vs. Post Checklist

	Pre-Checklist	Post-Checklist
Nerve Injuries Present on Pre-Operative Exam	28	33
Nerve Injuries NOT Present on Pre-Operative Exam	5	11
Nerve Injuries Present Post-Op Not Documented on Pre-Op Exam	7	0
Total Medial Pins Placed	23	14
Possible Iatrogenic Injuries Associated with Medial Pin Placement	2 (1 Radial, 1 Ulnar)	4 (1 Radial, 3 Ulnar)

In the post-checklist group, the number of medial pins placed (45) was significantly less than in the pre-checklist group (93) [p = 0.0001]. However, this finding was strongly influenced by faculty turnover (Figure 1). The behavior of the attending staff who were present both pre and post-checklist (C, D, E, F in Figure 1), did not appear to be influenced by the institution of the checklist, but this was not analyzed for significance.

FIGURE 1.

One faculty member left the institution just after the institution of the checklist (A) and two senior faculty members (B and C) stopped taking trauma call just after the institution of the checklist. Four faculty members began working at our institution only after implementation of the checklist (G, H, I, J). Of the attending staff whom were present both pre and post-checklist (C, D, E, F), there was no consistent trend towards a decrease in medial pin placement pre-and post-checklist.

In the pre-checklist group, the overwhelming majority (94%) did not have a specific reason cited within the operative report for medial pin placement. Post-checklist, no reason for medial pin placement was given in 18 cases, and ‘fracture instability’ was the second most oft-cited reason.

10 patients returned to the operating room for a second procedure in the pre-checklist group, whereas 4 in the post-checklist group had a return to the operating room. This finding was significant [p = 0.015], however, the reasons return to the operating room were not related to checklist parameters (Table 4). In the pre-checklist group, the returns to the OR for loss of alignment were due to a fall in the post-operative period, pins being placed through the fracture site, and inadequate fixation. In the post-checklist group, the returns for loss of alignment were secondary to poor position of immobilization and loss of fixation, respectively. All patients who were taken back to the operating room for irrigation and debridement had laterally placed pins. The data were not analyzed according to surgeon experience or assisting resident PGY-level.

Table 4.

Reasons for Return to OR

	Pre-Checklist	Post-Checklist
Buried Pin Removal	4	2
Irrigation and Debridement	3	0
Loss of Alignment Requiring Revision	3	2

The checklist compliance of the attending physicians was 85.85% and the residents were compliant 83.11% of the time. There were documented discrepancies between resident and attending checklists in 7.38% of all total checklists. Of those discrepancies, 41% of them were differences between the resident pre-operative neurologic exam and the attending pre-operative neurologic exam, as documented by the checklist. This finding could potentially indicate that there is evidence of evolving neurologic injuries, as the residents and attending staff document their checklists at different times.

DISCUSSION

We had hypothesized that our institutional patient safety checklist would standardize surgeon behavior in a way that would decrease complications, and therefore, improve patient safety. Our checklist for operative supracondylar humerus fractures, as currently designed, is not affecting patient care in the manner in which we had hoped.

On initial evaluation of our data, it appeared that we might have been affecting clinical practice in regards to returns to the operating room (OR) and the placement of medial pins. However, on further examination, it became clear that the data in regards to the medial pins was confounded by faculty turnover and variation. In addition, when the returns to OR were examined, it was apparent that the reasons for returns to the OR were not actually connected to factors that were regulated by the checklist.

One of the main recommendations from the AAOS CPG² that our checklist takes into account is the ‘limited’ recommendation for placement of 2-3 lateral pins in favor of medial or cross-pin configurations². Cross-pin configurations have greater biomechanical strength torsionally than certain, but not all, lateral pin configurations^{8, 9}. However, the increased risk of iatrogenic injury to the ulnar nerve¹⁰ is routinely cited as a reason to avoid medial pin placement despite its potential biomechanical advantages. While we were not seeking to decrease the usage of the medial pin with our checklist, and this paper’s purpose is not to recommend for or against medial pin usage, medial pin placement was an integral measure in terms of checklist effect and adherence.

Overall, after the checklist was instituted, 45 medial pins were placed in 42 elbows, for a compliance rate with the guideline recommendation of placing 2-3 lateral pins of 93%, or a guideline non-compliance rate of 7%. This compliance rate far exceeds the general practices of the members of POSNA, of which 309 were surveyed, and 30% reported using cross-pin configuration^{11, 12}, as opposed to our 7%. Following the analysis of the first 315 checklist patients, Goldberg reported that 19% had medial pins placed¹². The drop to 7% after the study period ended, with 537 patients included, does possibly show a trend towards decreasing medial pin placement, however, this cannot be proven to be attributable to the checklist given our lack of a true comparison group.

Regardless of institutional adherence to this specific guideline, and therefore, the checklist, we did not actually alter patient care or safety significantly, as is evidenced by the characterization of nerve injuries in the patients who had a medial pin. Of patients with a medial pin placed, there were 4 total presumably iatrogenic ulnar nerve injuries noted: 1 in the pre-checklist group and 3 in the post-checklist group. Proportionally, there was a higher rate of ulnar nerve injuries potentially associated with medial pin placement in the post-checklist group (1% vs. 6%). However, clinically, none of the potentially iatrogenic nerve injuries in either group caused any long-term morbidity or required any surgical intervention.

The second major guideline that our checklist was based on is the ‘moderate’ recommendation that all displaced (Type II and III, and displaced flexion) supracondylar humerus fractures undergo closed reduction with pin fixation². While this is part of the checklist for operative supracondylar humerus fractures, there is no such corollary for those Type II fractures treated non-operatively. Therefore, there is no way currently to ensure that this guideline is being followed, as Type II fractures that are treated non-operatively fall out of the checklist clinical pathway.

Relying on the AAOS CPG for creation of such checklists may be a handicap in creating a truly effective patient safety measure. Oetgen et al recently evaluated the CPG for Pediatric Femur Fractures¹³, and showed that the guideline is not affecting treatment practices for those fractures, and that the lack of higher-level evidence within the guideline may affect adherence to it.¹⁴ This is similar to our findings with supracondylar humerus fractures, in which we have an impressive quantity of literature on which to base these guidelines, but are lacking higher level of evidence studies. It is possible that our checklist would have been more clinically effective if it were based on higher levels of evidence rather than the aggregate guideline that has not been updated since its inception.

The process of improving patient safety via a checklist also involves implementing a system that can and will be used consistently by the responsible physicians. From a systems standpoint, our study demonstrated that a patient safety checklist can be implemented in a large tertiary care center with reasonable success in terms of compliance, as previously posited by Goldberg¹². For our checklist system specifically, the compliance could be improved with adjustments to the notification system for attending staff, more stringent requirements in regards to the timing of checklist attestation, and disabling the ability to simply close a checklist window without completing it. Adjusting and making the checklist a more fluid protocol may allow our outcomes to mirror the success of other validated checklist systems.

The study of our own checklist was not without its limitations. This checklist is institution-specific and within its first iteration during the study period. The study was also limited by the number of surgeries performed at our center, and by the reasonable time-span to look before and after the institution of the checklist. Despite methodological imperfections, our study allowed us to share our challenges with development of a patient safety tool.

The WHO Surgical Safety Checklist (SSC) continues to serves as a model for a patient safety metric in the surgical arena. The success of this checklist has driven the development of many other checklists, including our own, in the hopes that they will enable further quality improvement. The outcomes of the large-scale implementation of the WHO SSC was studied as part of the WHO’s Safe Surgery Saves Lives Program between 2007 and 2008, and was shown to significantly decrease rates of mortality as well as inpatient complication.¹⁵ This particular checklist has been very well-studied across many surgical subspecialties as well as the field of anesthesia with a recent meta-analysis, counting 723 studies in total, showing the implementation of the SSC to be associated with a reduction in post-operative complications and mortality.¹⁶ The success of this process has even permeated mainstream literature in Atul Gawande’s Checklist Manifesto. Given the documented improvements using the SSC, it is tempting to assume that this this type of process will work for many and varied situations, and to apply it to as many arenas as possible, as we did at our institution. However, our current research highlights the fact that the institution of checklist protocol for certain surgical procedures and diagnoses does not always result in clinical change or improvement in patient safety.

The effectiveness of measures such as safety checklists can be diminished and ‘checklist fatigue’ can result as physicians become bombarded by computer-based quality improvement measures that become attached to nearly every clinical activity. This is particularly true for surgical subspecialties in which a checklist can be seen as relegating surgical decision-making to checking boxes on a formulary. The autonomy, creativity, and comfort of the surgeon can be threatened by the presence of endless checklist protocols. As Grigg stated in 2015, “Checklists need to be seamless, to minimize inconvenience and ensure compliance, yet stimulating enough to not become mindless and lose meaning.”¹⁷ They need to evolve along with medical and surgical practices, and along with physicians and surgeons. While our faculty, residents, and ancillary staff were not surveyed or questioned in regards to ‘checklist fatigue’ at the time of this study, this may become an area of future research for our institution.

The findings of this study do not by any means indicate that we should dismiss the checklist as a means of quality improvement, but we may need to be more discriminating in regards to its applications and discerning in its creation.