Classifying Adverse Events Following Lower Limb Orthopaedic Surgery in Children With Cerebral Palsy: Reliability of the Modified Clavien-Dindo System

Supplemental Digital Content is available in the text.

Abstract

Background:

The modified Clavien-Dindo (MCD) system is a reliable tool for classifying adverse events (AEs) in hip preservation surgery and has since been utilized in studies involving lower limb surgery for ambulant and nonambulant children with cerebral palsy (CP). However, the profile of AEs recorded in children with CP compared with typically developing children is different, and the reliability of the MCD in CP is unknown. This study aimed to evaluate the interrater and intrarater reliability of the MCD system for classifying AEs following lower limb surgery in children with CP.

Methods:

Eighteen raters were invited to participate, including clinicians from surgical, nursing, and physical therapy professions, and individuals with CP. Following a MCD familiarization session, participants rated 40 clinical scenarios on 2 occasions, 2 weeks apart. Fleiss’ κ statistics were used to calculate interrater and intrarater reliability.

Results:

The overall Fleiss’ κ value for interrater reliability in the first rating was 0.70 (95% confidence interval, 0.61-0.80), and increased to 0.75 (95% confidence interval, 0.66-0.84) in the second rating. The average Fleiss’ κ value for intrarater reliability was 0.78 (range, 0.48 to 1.00). Grading of more severe AEs (MCD III to V) achieved near perfect agreement (κ, 0.87 to 1.00). There was a lower level of agreement for minor AEs (MCD I-II) (κ, 0.53 to 0.55). A κ score of 0 to 0.2 was deemed as poor, 0.21 to 0.4 as fair, 0.41 to 0.6 as good, 0.61 to 0.8 as very good, and 0.81 to 1.0 as almost perfect agreement.

Conclusions:

The MCD System demonstrates a very good interrater and intrarater reliability following lower limb surgery in children with CP. The MCD can be used by clinicians from different health care professions with a high level of reliability. The MCD may improve standardization of AE recording with a view to accurate audits and improved clarity in outcome studies for CP.

Level of Evidence:

Level II—diagnostic.

Reporting surgical adverse events (AEs) is a critical component of surgical outcome studies but does not always receive adequate methodological rigor. Just over 20 years ago, the editor of the Lancet identified poor study design and lack of standardization in the reporting of safety and outcomes relevant to patients, as areas of major concern in surgical literature.1

In an effort to address this, Clavien and Dindo developed a valid and reliable classification system for AEs in general surgery,2–4 which is perceived similarly by patients, nurses, and physicians.5 In its first orthopaedic application, Sink et al6 modified the Clavien-Dindo system (MCD) for hip preservation surgery and showed good interrater and intrarater reliability. Until now, there has been no standardized system for reporting the severity of AEs in orthopaedic surgery and little is known about the reliability of reporting AEs in complex neurological conditions, such as cerebral palsy (CP).6,7

A retrospective matched cohort study by DiFazio et al8 utilized the MCD for reporting postoperative complications after hip surgery in nonambulant children with CP compared with children without CP. The study showed that children with CP had more frequent AEs than children without CP, and had a different profile of AEs.8 In children without CP, surgical AEs such as delayed union, wound hematomas and neuropraxias were the most common complications. In children with CP, skin complications were frequent as were medical AEs including gastrointestinal and respiratory complications.8 Most recently, Dreher et al9 utilized the MCD in a long-term multicenter retrospective study, reporting the outcomes of surgery to improve gait in ambulant children with CP. Nearly half of the children suffered from an AE, most of which were self-limiting. The use of the MCD as a formal tool to report surgical AEs was commended by Theologis,10 in a commentary for the study by Dreher et al.9

We have also found the MCD by Sink et al6 to be useful for reporting AEs following orthopaedic surgery in children with CP. Given that the AE profiles of children with CP and typically developing children are very different, further minor modifications and clarifications appropriate for use in children with CP were considered to be required,8 along with evaluating the reliability of this modified MCD for use in children with CP.

The aim of the study was to evaluate the interrater and intrarater reliability of the MCD of AEs for use in CP after lower limb orthopaedic surgery, and to determine if the MCD was a tool that could be easily used by medical and health professionals in a multidisciplinary setting, and be understood and acceptable to individuals with CP and their families.

METHODS

Developing a MCD System for CP

The MCD is a 5-grade ordinal system (Table 1), with grading determined by the treatment required to manage the AE and any associated long-term morbidity. We based our MCD on modifications made by Sink et al6 to the CD, in combination with the profile of AEs experienced by children with CP after orthopaedic surgery in recently reported studies.7,8 Using information from these studies, we made several customizations to the MCD to increase its applicability to children with CP (Table 1, Appendix 1, Supplemental Digital Content 1, http://links.lww.com/BPO/A170).

TABLE 1.

Summary of Definitions of Each Grade of the Modified Clavien-Dindo System, Modified for Children With Cerebral Palsy, including examples for each grade

Participants

We identified and invited 18 multidisciplinary members who worked closely within a team and were willing to participate in the ratings of 40 clinical scenarios related to children with CP who had undergone orthopaedic surgery at a single tertiary pediatric center. The panel of raters comprised of the following:

• Eight orthopaedic surgeons (3 attendings with fellowship training in pediatric orthopaedics, 3 clinical fellows in pediatric orthopaedics, and 2 research fellows).

• Five senior physical therapists who were working in a gait laboratory or hip surveillance service.

• Two senior clinical nurse coordinators, with both ward and outpatient experience.

• Three young adults with CP who had previously experienced orthopaedic surgery and who were currently working in the medical field.

Reliability Ratings

We drew upon the experience of Sink et al6 to determine the number of clinical scenarios required for the study. Forty clinical scenarios were created from complications recorded over a 20-year period in our tertiary center, which provides surgical care for a large population of children with CP. Scenarios were aimed to reflect a combination of recently published AEs.7–9 Each scenario was based on a specific patient and real clinical events with a minimum 2-year follow-up to be certain about long-term outcomes. Information included age, sex, Gross Motor Function Classification System level, clinical and operative information, details about the AE, management of the AE, and long-term outcome. Clinical photographs or radiographs, which had been archived from the time of the AE, were used to illustrate each scenario. The clinical scenarios were presented in PowerPoint (Appendix 2, Supplemental Digital Content 2, http://links.lww.com/BPO/A171). All MCD grades were represented across the scenarios, but they did not reflect the frequency of MCD grades from previous studies (Fig. 1).7,8 Patient data was deidentified and institutional privacy regulations observed.

FIGURE 1.

Distribution of graded MCD adverse events used for clinical scenario ratings. MCD indicates modified Clavien-Dindo system.

Immediately before the first rating, a meeting was held with all raters to provide orientation to the MCD. All raters were presented with material to familiarize them with the MCD for CP, including a detailed description of the study purpose and the MCD (Appendix 3, Supplemental Digital Content 3, http://links.lww.com/BPO/A172). Four example scenarios were provided for each MCD grade and discussed. Using skin-related AEs as a specific example, a priori discussion with raters agreed on the following:

• Grade I—skin redness or irritation from a cast or splint which is resolved by a simple adjustment.

• Grade II—a partial thickness skin breakdown which required outpatient dressings.

• Grade III—the presence of a full thickness ulcer, requiring surgical debridement, skin grafting, or a skin flap, under general anesthesia.

The first ratings were completed following the familiarization meeting. The second rating was conducted 2 weeks later, with prior randomization of the scenario presentation order. Raters recorded their responses on a standard form and responses were then collated and transcribed onto an Excel spreadsheet. Each rating was identified by the rater’s profession and rating occasion, (eg, PTR1=physical therapist rating 1), but individuals were not identified.

Statistical Analyses

The data was analyzed with statistical support provided external to our institution. Fleiss’ κ statistics with 95% confidence intervals (CI) were used to determine the level of agreement (interrater reliability) between (Tables 2 and 3)

1. All raters, across all MCD grades.

2. Professional group across all MCD grades.

TABLE 2.

Fleiss’ κ for Interrater Reliability of the Modified Clavien-Dindo System for Children With Cerebral Palsy

TABLE 3.

Fleiss’ κ for Interrater Reliability Within Each Modified Clavien-Dindo System Grade Among All Raters

Fleiss’ κ statistics with 95% CI were also used to describe the intrarater reliability for each rater (Table 4). The preference to use Fleiss’ κ over weighted κ were 2-fold: to eliminate the risk of falsely inflating our results, and to increase result reproducibility.

TABLE 4.

Fleiss’ κ for Intrarater Reliability of the Modified Clavien-Dindo System for Children With Cerebral Palsy

A κ score of 0 to 0.2 was deemed as poor, 0.21 to 0.4 as fair, 0.41 to 0.6 as good, 0.61 to 0.8 as very good, and 0.81 to 1.0 as almost perfect agreement.12,13 All calculations were performed in Stata version 14.1 (StataCorp, College Station, TX).

RESULTS

Rating 1 Interrater Reliability

The overall agreement for all raters on all MCD grades was very good (κ, 0.70; 95% CI, 0.61-0.80) (Table 2). The overall agreement varied among professional groups, ranging from κ 0.59 to 0.93. A difference in scores was noted between MCD I and II, compared with agreement among the other grades (Table 3). Overall agreement for MCD I and II was good (κ: 0.52, 0.47, respectively). Grading of MCD III, IV, and V AEs were near perfect (κ: 0.84, 0.97, 1.00, respectively).

Rating 2 Interrater Reliability

Reliability increased at rating 2, with very good overall agreement on MCD grades (κ, 0.75; 95% CI, 0.66-0.84) (Table 2). Grading of MCD I and II AEs slightly improved but remained good (κ: 0.57, 0.58, respectively) (Table 3). Grading of MCD III, IV, and V AEs were once again near perfect (κ: 0.90, 0.95, 1.00, respectively).

Intrarater Reliability

Overall intrarater reliability among raters was very good (average κ, 0.78). The highest level of agreement was seen among attending surgeons (κ, 0.80 to 1.00), orthopaedic clinical fellows (κ 0.71 to 0.90), and individuals with CP (κ, 0.61 to 0.93) (Table 4). The lowest level of agreement was observed among physical therapists (κ, 0.48 to 0.80).

DISCUSSION

Preventing surgical AEs is critically important to providing high-quality clinical care, minimizing harm, maximizing function, and containing health care costs.8,14,15 Historically, surgeons have reported short-term complications and longer term treatment outcomes that do not involve patients’ perceptions16–19 and is hindered by a lack of standardization and reproducibility.20 A classification system that is valid, reliable, and easy to use may permit transparent standardized reporting of AEs, improve the accuracy of audits, and lead to more objective understanding and comparison of surgical outcome studies in the literature. Use of such systems can promote early recognition of events that deviate from the normal postoperative course, to prevent a cascade effect that could cause permanent morbidity or mortality. Clear documentation and collection of data related to complications will also add information on the associated risks of surgery, guiding the shared or informed decision-making process with parents, caregivers, and young people with CP.7,8

This study shows a very good interrater and intrarater reliability of the MCD for lower limb surgery in children with CP. The system has good face validity and we have shown that it can be used reliably in a multidisciplinary team environment by surgeons, allied health and nursing professionals. Our data shows that a high level of agreement was consistent among orthopaedic attending surgeons and clinical fellows, which could reflect their experience and expertise in recognizing and managing AEs.

Raters had lower agreement with MCD grades I and II overall, and this may indicate some uncertainty in distinguishing typical postoperative sequelae from minor AEs. Subtle differences exist between a treatment considered “preexisting” compared with the addition of new treatment. We think that agreement between MCD grades I and II could be improved by a priori discussion, and the development of more detailed guidelines are now included in Appendix 1 (Supplemental Digital Content 1, http://links.lww.com/BPO/A170).

Near perfect agreement was achieved among MCD grades III, IV, and V, irrespective of professional background. This suggests that grading by treatments required to address the AE was clearly understood, and that the outcome worsened with increasing MCD grade.

Children who are most in need of major reconstructive lower limb surgery21–23 are often the most medically frail.8,18 MCD grades I and II events occurred in up to 60% of children in a recent prospective cohort study of hip surgery in nonambulant children with CP.7 The most common complications were constipation, cast-related or splint-related skin irritation, inadequate pain and spasm control from malfunctioning epidural or morphine infusions, and respiratory infections. Most were self-limiting or resolved with simple medical treatment.7 However, some children had multiple AEs and in other children, minor AEs escalated to more serious events. For example, children functioning at Gross Motor Function Classification System V have a high prevalence of preexisting respiratory disease. Attempts to manage postoperative pain and spasm with narcotic infusions and diazepam can lead to respiratory depression and pneumonia, which may require an intensive care unit admission and mechanical ventilation.7,8 DiFazio et al8 reported similar findings, in which 65% of children with CP suffered from postoperative AEs at a rate nearly twice that of typically developing children. This contrasts with a 10.5% surgical and a 29.8% medical AE reported in a large, retrospective cohort study of 168 hip reconstructions involving 121 children with CP.22 Constipation and inadequate analgesia were not reported, which may be due to the retrospective nature of the study.22

The difference between the reported rates of AEs in outcome studies of children with CP suggests that underreporting of AEs exists in the literature. This is an important issue that requires addressing. Surgeons readily accept that wound infections, hardware failure, or nonunion are surgical AEs that should be reported.6,8 As physicians who operate, surgeons should also take responsibility for reporting, preventing, and ameliorating “medical” AEs. If the child was not subjected to general anesthesia and the operation, the exacerbation of constipation, pain, hypertonia, aspiration, and chest infection would not have occurred. Medical and surgical AEs should be “owned,” prevented, and managed by all members of the clinical team. This is the principal reason that we involved all members of our multidisciplinary team in assessing the reliability of the MCD for CP, as well as the perspective of individuals with CP. We strongly encourage a proactive system of prospective documentation, to improve the transparency of recognizing and reporting events.

The key strengths of our study include the inclusion of participants from medical, nursing, and allied health professions, and of individuals with CP whom have experienced orthopaedic surgery. The diversity of our raters aimed to model holistic care provided by a multidisciplinary team, and included the perceptions of AEs in individuals with CP.

This study is the first to examine the reliability of the MCD in children with CP, and it has some limitations. First, raters were inexperienced in using the MCD, resulting in less than perfect agreement. Second, this study highlights the difficulty of distinguishing the severity of minor AEs (MCD I and II), with little comparative data in existing literature. With increasing utilization of this system in time, it is felt that reliability in grading minor AEs will also continue to improve, shown by the increased observed agreement from rating 1 to rating 2. Future reporting of all minor AEs should be encouraged in the literature to characterize the true frequency of events, identify risk factors and develop strategies for prevention in the postoperative period. Third, many of our complex care children experience more than 1 AE. The MCD system currently has no guidance on how to rate or summarize the effects of multiple complications.

In conclusion, this study has shown that the MCD is a system that can be utilized reliably for grading AEs in CP, following hip and lower limb surgery. The MCD is a useful tool that is easily understood and can be explained to any member of the multidisciplinary team. Future research will be directed on developing a system for grading multiple AEs and testing the validity of the MCD in lower limb surgery for CP in a real time, prospective clinical trial. We recommend the use of the MCD until there is consensus for a gold standard system in AE reporting.

Supplementary Material

Supplemental Digital Content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's website, www.pedorthopaedics.com.