Spillover of Early Extubation Practices from the Pediatric Heart Network Collaborative Learning Study

Abstract

Objective

The Pediatric Heart Network Collaborative Learning Study (PHN CLS) used collaborative learning strategies to implement a clinical practice guideline (CPG) that increased rates of early extubation (EE) after infant repair of tetralogy of Fallot and coarctation of the aorta. We assessed EE rates for infants undergoing cardiac surgeries not targeted by the CPG to determine whether changes in extubation practices spilled over to care of other infants.

Design

Observational analyses of site’s local Society of Thoracic Surgeons Congenital Heart Surgery Database (STS-CHSD) and Pediatric Cardiac Critical Care Consortium (PC4) Registry

Setting

Four PHN CLS active site hospitals

Patients

Infants undergoing ventricular septal defect repair, atrioventricular septal defect repair, or superior cavopulmonary anastomosis (lower complexity), and arterial switch operation or isolated aortopulmonary shunt (higher complexity)

Interventions

None

Measurements and Main Results

Aggregate outcomes were compared between the 12 months pre-CPG and 12 months after study completion (Follow Up). In infants undergoing lower complexity surgeries, EE increased during Follow Up compared with Pre CPG (30.2% vs 18.8%, p = .006), and hours to initial postoperative extubation decreased. We observed variation in these outcomes by surgery type, with only VSD repair associated with a significant increase in EE during Follow Up compared with Pre CPG (47% vs 26%, p = 0.006). Variation by study site was also seen, with only one hospital showing an increase in EE. In patients undergoing higher complexity surgeries there was no difference in EE or hours to initial extubation between the study eras.

Conclusions

We observed spillover of extubation practices promoted by the CLS CPG to lower complexity operations not included in the original study that was sustainable 1 year after study completion, though this effect differed across sites and operation subtypes. No changes in postoperative extubation outcomes following higher complexity surgeries were seen. The significant variation in outcomes by site suggests that center specific factors may have influenced spillover of CPG practices.

Introduction

The Pediatric Heart Network Collaborative Learning Study (PHN CLS) demonstrated that a clinical practice guideline (CPG) developed using collaborative learning principles could be successfully implemented across centers, and lead to increased rates of early extubation (EE) in infants following repair of tetralogy of Fallot and coarctation of the aorta, from 12% to 67% at active study sites (p <0.001) compared with no change at control sites (1). In order to fully understand the impact of quality initiatives like this, it is necessary to consider not only the intended changes in care practices, but also the unintended influences of the intervention on other aspects of care.

Broadly defined, spillover is a process by which activity in one setting has wider impact on activities in other, unrelated settings through overflow of concepts and knowledge. In the context of clinical medicine, spillover is said to occur when an intervention targeted to one aspect of care leads to changes in other care practices, or in other patient groups. As an example, in a randomized controlled trial of automated reminders for intraoperative antibiotic redosing during prolonged cardiac surgery, the intervention led to improved intraoperative and postoperative dosing and reduced postoperative wound infections in both intervention and control cohorts (2). This illustrates beneficial spillover beyond the scope of an intervention to other elements of care and to other patients, and shows that failure to account for spillover can result in under estimation of the treatment effect of an intervention.

In order to more completely assess the impact of the CLS, we evaluated whether practice changes stemming from the CPG impacted care outside of the CLS population. Our specific objective was to determine whether changes in extubation practices promoted by the CLS spilled over to the care of infants undergoing cardiac surgical procedures not included in the primary study at the 4 active sites.

Materials and Methods

Eras of comparison and data sources

The methods and main results from The PHN CLS have been previously published (1, 3). Briefly, an investigative team observed variation in postoperative mechanical ventilation practices and clinical outcomes after infant surgery at five congenital heart surgery hospitals that were all core PHN sites. One hospital was identified as a positive outlier, with much lower median ventilation times and shorter lengths of stay (LOS) after infant repair of tetralogy of Fallot and coarctation of the aorta. Participants engaged in a series of round-robin site visits and then created a CPG based on extubation strategies and associated key practices at this “model” site. In the main study, clinical outcomes at the four intervention hospitals that implemented the CPG were measured during the 12 months before (pre-CPG) and the 12 months after (post- CPG) implementation and were compared with the five other PHN sites during the same time period (controls) who continued with usual practice. Significant increases in EE rates were seen at intervention hospitals following CPG implementation, but not at the control sites that did not use the CPG.

To evaluate the presence of spillover of care practices promoted by the CLS CPG to patients not included in the primary study we selected time periods for comparison based on those used in CLS (Figure 1). The Pre CPG era was the 12 month period immediately preceding the implementation of the PHN CLS CPG, and the Follow Up era was the 12 month period beginning 3 months after completion of the CLS. Our purpose in selecting for the Follow up era the 12 month period beginning 3 months after completion of the CLS was two-fold; first, we wanted to examine sustainability of any spillover of care practices that we observed, as results from a prior study showed variable sustainability of outcomes from the main CLS (4). The second, more pragmatic, reason was that PC4 was not in use at all study sites until this time period, and utilizing this registry made data collection for the Follow Up era more uniform and efficient.

Figure 1.

Eras of comparison. The Collaborative Learning Study clinical practice guideline was utilized at active sites from April 2014 through March 2015

We used existing clinical registry data for study outcomes. Information from each site’s local surgical database collected for submission to The Society of Thoracic Surgeons Congenital Heart Surgery Database (STS-CHSD) was used to obtain data for the Pre CPG era, supplemented by retrospective review of patients’ health records at the active sites as needed to capture all courses of mechanical ventilation and ICU admissions. Data for the Follow Up era was obtained in entirety from the Pediatric Cardiac Critical Care Consortium (PC4) Registry, as during this era all study sites were fully participating in this registry. Both datasets share common coding and definitions for peri-operative variables, and published audit results suggest a high degree of data completeness and accuracy (5, 6).

Patient Population

We selected three cardiac surgeries of lower complexity, for which early postoperative extubation would be common, including repair of ventricular septal defect (VSD), repair of complete atrioventricular septal defect (AVSD), and superior cavopulmonary anastomosis (SCPA), including Glenn and hemi Fontan operations. We also wanted to evaluate changes in extubation practices in infants undergoing operations for which early postoperative extubation would not be usual practice, to assess whether spillover of early extubation would be limited by surgical complexity, yet translate to changes in extubation practices outside of the early extubation period. This higher complexity cohort included infants undergoing arterial switch operation (ASO), with or without VSD closure, and isolated aortopulmonary shunt (APS) placement.

Eligible patients were those undergoing one of the selected operations at < 365 days of age at one of the 4 active sites from the CLS. To decrease likelihood of co-morbidites that might impact EE but not be captured from retrospective chart review or registry data, patients were excluded if they were hospitalized for more than 1 day immediately preceding a lower complexity surgery. Patients were excluded from the VSD closure or APS placement cohorts if they underwent these procedures as part of a more complex operation.

Outcome Variables

The primary outcome for the lower complexity surgery cohort was EE rate, defined as percentage of eligible patients who were extubated in the operating room (OR) or within 6 hours of postoperative ICU admission. Secondary outcomes for the lower complexity cohort included hours to initial postoperative extubation, rate of reintubation within 48 hours of initial extubation, and postoperative ICU and hospital lengths of stay (LOS).

For the higher complexity surgery cohort we selected hours to initial postoperative extubation as the primary outcome, as EE seemed unlikely in this group. Secondary outcomes were EE rate, rate of reintubation within 48 hours of initial extubation, and postoperative ICU and hospital LOS.

Data for the Pre CPG era were uploaded from each site’s local surgical database collected for submission to the STS-CHSD into REDCap (Research Electronic Data Capture) (7), hosted at the University of Michigan, and data for the Follow Up period were extracted from the PC4 registry. All analyses were performed at the PC4 Data Coordinating Center at the University of Michigan. The University of Michigan Institutional Review Board provides oversight for the PC4 Data Coordinating Center, and data collection for the Follow up period was approved with waiver of informed consent. Amendments to the approved IRB application for the main PHN CLS were used to obtain approval with waiver of consent at individual sites for collection of patient data for the Pre CPG period.

Statistical Analysis

All analyses across eras were presented as frequencies or percentages for categorical variables and as medians with interquartile ranges for continuous variables. We evaluated variables in univariate analyses using chi-square test, Fisher exact test for categorical variables, and Wilcoxon rank sum test for continuous variables, as appropriate.

Primary and secondary outcomes were analyzed for patients in aggregate, and across study site, operation type and operation complexity as defined in the preceding section. Analyses were performed using SAS Version 9.4 (SAS Institute, Cary, NC) or STATA Version 14 (Stata Corp, College Station, TX).

Results

Patient Characteristics

There were 351 infants meeting eligibility criteria in the Pre CPG era, and 360 in the Follow Up period. There was no significant difference in the proportion of lower vs higher complexity surgeries between the time periods, nor in the distribution of surgery subtypes. (Table 1). There were no significant differences in age or weight at surgery, gender distribution or incidence of preterm birth between the eras (Table 2). There was no difference between the cohorts in percentage of patients with Trisomy 21 (20% Pre CPG vs 22% Follow Up, p = .40).

Table 1.

Distribution of Surgery Types Across Study Eras

Surgery type, n (%)	Pre CPG Era (n = 351)	Follow Up Era (n = 360)
Lower Complexity	213 (61%)	225 (62%)
VSD	87 (25%)	74 (21%)
CAVSD	53 (15%)	66 (18%)
SCPA	73 (21%)	85 (24%)
Higher Complexity	138 (39%)	135 (38%)
Isolated APS	63 (18%)	66 (18%)
Arterial switch	54 (15%)	45 (13%)
Arterial switch + VSD	21 (6%)	24 (7%)

VSD = ventricular septal defect, CAVSD = complete atrioventricular septal defect, SCPA = superior cavopulmonary anastomosis, APS = aortopulmonary shunt

Table 2.

Patient Characteristics in the Pre CPG and Follow up Study Eras

Patient characteristic	Pre CPG (n = 351)	Follow Up (n = 360)	p value
Age at surgery, days, Median (IQR)  Overall	125 (9–179)	132 (10–186)	0.57
Lower complexity surgeries	211 (29–347)	212 (51–350)	0.34
Higher complexity surgeries	14 (0–238)	15 (1–339)	0.78
Weight at surgery, kg, Median (IQR)  Overall	5 (3.4–6.1)	5 (3.5–6.2)	0.30
Lower complexity surgeries	5.8 (5.0 – 6.6)	6 (5.2–6.8)	0.26
Higher complexity surgeries	3.2 (2.8–3.7)	3.3 (2.9–3.7)	0.56
Preterm birth (< 37 weeks), n (%)  Overall	57 (16.2)	73 (20.3)	0.16
Lower complexity surgeries	36 (17)	49 (21.8)	0.20
Higher complexity surgeries	21 (15.2)	24 (17.8)	0.57

IQR = interquartile range, kg = kilogram

Lower Complexity Cohort

The results for the primary outcome in the lower complexity surgery cohort, EE rate, are shown in Figure 2. In aggregate, the EE rate increased from 19% in the Pre CPG era to 30% in the Follow Up period (p = 0.006). This effect was largely driven by patients undergoing VSD repair, in whom EE increased from 26% to 47% (p = 0.006). More modest, nonsignificant increases in EE were seen in the other lower complexity subtypes (Figure 2a).

Figure 2.

Early extubation rates in the lower complexity surgery cohort, in aggregate and by surgery subtype (2a), and by study site (2b).

When analyzed by site, notable heterogeneity in outcomes existed, both in EE rates during the Pre CPG period, and in the change in EE between the two eras (Figure 2b). Only Site A demonstrated a significant increase in proportion of patients extubated early, from 14% to 56% (p < 0.0001). Site D had a relatively high rate of EE in the Pre CPG period which did not change significantly between the eras.

The results for one of our secondary outcomes for the lower complexity surgery cohort, median hours (IQR) to initial postoperative extubation, are shown in Figure 3. Time to initial extubation for the overall cohort decreased significantly from 17 hours (8 – 26) to 12 hours (3 – 23), p = 0.007. This effect is again driven by patients undergoing VSD repair, in whom time to initial extubation decreased from 14 hours (5 – 25) to 6 hours (0 – 14), p = 0.0002. No significant changes in hours to initial extubation were seen in the other surgery subtypes (Figure 3a).

Figure 3.

Median hours to initial postoperative extubation in the lower complexity surgery cohort, in aggregate and by surgery subtype (3a) and by study site (3b).

When analyzed across study sites, considerable variation in median time to initial postop extubation was observed (Figure 3b). Significant changes in time to initial extubation were seen only at Site A, where it decreased from 31 hours (13 – 73) to 3 hours (0 – 46) p = 0.0004, and Site B, where was a decrease from 18 hours (10 – 24) to 11 hours (4 – 20), p < 0.05.

There were no significant differences between Pre CPG and Follow Up eras for our other secondary outcomes of rate of reintubation and LOS for the lower complexity surgery cohort as a whole (Table 3). However, when analyzed by surgery subtype, postoperative ICU LOS decreased significantly only in patients undergoing VSD repair. Analysis by site showed a significant decrease in ICU LOS only at Site A, while Site C showed a small but significant increase in postoperative hospital LOS.

Table 3.

Secondary Outcomes for Lower Complexity Surgery Cohort

Surgery and Study Site	Reintubation within 48 hrs extubation n (%)		ICU Length of Stay Median (IQR) days		Hospital Length of Stay Median (IQR) days
	Pre CPG	Follow Up	Pre CPG	Follow Up	Pre CPG	Follow Up
Overall Cohort	8 (3.8)	13 (5.8)	2.8 (1.9–2.3)	2.1 (1.8–4.1)	6 (4–8)	6 (4–10)
VSD	3 (3.4)	2 (2.7)	2.1 (1.3–3.4)	1.3 (1.1–2.2) a	5 (4–7)	5 (3–6)
CAVSD	2 (3.8)	7 (10.6)	3.0 (2.1–5.0)	2.8 (2.0–5.1)	7 (5–9)	7 (5–11)
SCPA	3 (4.1)	4 (4.7)	3.0 (1.9–5.0)	2.8 (2.0–5.0)	7 (5–14)	7 (5–10)
By Site
Site A	6 (10.7)	5 (11.6)	4.2 (3.0–5.9)	2.0 (1.1–5.9) a	9 (6–16)	9 (5–11)
Site B	0 (0)	2 (4.1)	3.8 (2.8–4.9)	2.8 (1.9–5.0)	7 (6–8)	6 (5–10)
Site C	2 (2.6)	4 (4.4)	2.0 (1.3–3.2)	2.1 (1.9–3.9)	4 (4–6)	5 (4–8) b
Site D	0 (0)	2 (4.8)	1.8 (1.1–2.2)	2.1 (1.2–4.0)	5 (4–6)	5 (4–7)

IQR = interquartile range, VSD = ventricular septal defect, CAVSD = complete atrioventricular septal defect, SCPA = superior cavopulmonary anastomosis,

^ap <0.005

^bp< 0.05

Higher Complexity Cohort

No significant changes in extubation outcomes between study eras were demonstrated in the higher complexity surgery cohort (Table 4). Time to initial postoperative extubation was longer than in the lower complexity group, and the rate of reintubation within 48 hours of initial extubation was greater in this higher complexity group. Only 2 patients in the Pre CPG cohort and none in the Follow Up period were extubated within 6 hours of ICU admission.

Table 4.

Extubation Outcomes for Higher Complexity Surgery Cohort

Extubation outcome	Pre CPG Era ( n = 138)	Follow Up Era (n = 135)	p value
Hours to initial extubation, Median (IQR)  Overall	71 (41–117)	73 (47–116)	0.45
Aortopulmonary shunt	53 (26–113)	69 (30–114)	0.64
Arterial switch	72 (44–97)	77 (62–116)	0.47
Arterial switch + VSD repair	99 (67–133)	90 (66–128)	0.74
Early Extubation, n (%)  Overall	2 (1)	0 (0)
Aortopulmonary Shunt	0 (0)	0 (0)
Arterial switch	1 (2)	0 (0)
Arterial switch + VSD repair	1 (5)	0 (0)
Reintubation rate, n (%)  Overall	12 (8.7)	10 (7.4)	0.70
Aortopulmonary Shunt	4 (6.4 )	4 (6.1)	0.99
Arterial switch	5 (9.3)	3 (6.7)	0.72
Arterial switch + VSD repair	3 (14.3)	3 (12.5)	0.99

IQR = interquartile range, VSD = ventricular septal defect

Discussion

The concept of spillover is frequently described in the contexts of economies and human behavior; however, there is evidence for the impact of this phenomenon in clinical medicine in general, and in clinical guideline implementation in particular. For example, the introduction of clinical guidelines for asthma care has been associated with improved diabetes outcomes, presumably through sensitization of clinicians to use of guidelines (8), and clinical guidelines targeted at improving breast cancer screening have been associated with improved rates of cervical cancer detection (9). While most instances of spillover in clinical medicine lead to benefit, spillover of clinical practice guidelines to patient groups for whom they were not designed can also result in undesirable outcomes. Spillover effects of an FDA warning about risk of suicidality in pediatric patients treated with antidepressants were associated with a decline in appropriate diagnosis and treatment among adults with depression (10).

Given the potential for both beneficial and detrimental consequences, accounting for spillover effect is necessary for understanding the true impact and appropriate use of clinical interventions like the PHN CLS early extubation CPG. Failure to recognize beneficial spillover may lead to underestimation of the value of an intervention, as differences between treatment and control groups may be minimized by spillover. Similarly, the harmful effects of an intervention when it results in changes in care in populations it was not intended to impact must also be examined in order to guide appropriate utilization going forward.

We observed positive spillover of early extubation practices from the CLS CPG to infants undergoing lower complexity surgeries that was sustainable after completion of the CLS. There were statistically significant changes in both EE rate and in time to initial postoperative extubation for this cohort as a whole, although this appeared to be driven primarily by patients undergoing VSD repair. Surgical complexity, as measured by Society of Thoracic Surgeons-European Association of Cardiothoracic Surgery (STAT) Mortality Categories, among other factors, has been associated with duration of mechanical ventilation after neonatal cardiac operations (11). Thus it is not surprising that differences in the spillover effect of EE across surgery subtypes in our study appeared to be related to surgical complexity; we observed no changes in extubation outcomes in the higher complexity surgery cohort. The CLS CPG was focused on intraoperative anesthesia and early postoperative analgesia and sedation, and as these specific practices are less likely to impact the care of patients who typically require support with mechanical ventilation beyond the early postoperative period, the lack of spillover of CPG care practices to the higher complexity surgery cohort is understandable.

In the lower complexity cohort, extubation practices following VSD repair changed the most following CLS CPG implementation, and it is likely that adopting EE in patients with biventricular circulation undergoing corrective surgery was more feasible than in the other surgery subtypes. The favorable hemodynamic effects of spontaneous ventilation when pulmonary blood flow is dependent on intrathoracic pressure are well established, and EE in these patients has been associated with improved outcomes (12–14). This knowledge may have led to earlier extubation in low risk patients undergoing SCPA prior to the PHN CLS, and thus opportunity for further reduction in intubation times related to CPG spillover may have been limited. Likewise, Trisomy 21 has been identified as a strong predictor of delayed extubation following heart surgery, and given the very high rate of chromosomal abnormalities in infants with AVSD, it is possible that extra-cardiac comorbidities may have deterred spillover of EE practices in this surgery cohort (15,16).

In the present study there was also heterogeneity in outcomes across sites, with only one site demonstrating a significant change in EE rate from Pre CPG to Follow Up era. Variation in clinical practices and outcomes across centers is widely recognized, and observed differences in extubation outcomes after pediatric cardiac surgery across PHN sites was a major impetus for the CLS (1, 17). Bates and colleagues showed that while all 4 active sites in the PHN CLS significantly reduced time to extubation, there was notable variation across sites in the Pre CPG median time to extubation (range 15.4 – 35.5 hours) and also in the magnitude of change in time to extubation following CPG implementation (- 73.3 to −99.2%) (18). Differences between PHN CLS active sites in the strategies that were utilized for CPG implementation were also noted, and appeared to have influenced the impact of the CPG at different sites. The site that utilized the most unique implementation strategy (Site A), which included development of complementary care protocols and a more inclusive, multidisciplinary team approach, realized the shortest post CPG time to extubation and had the greatest percent change.

In another ancillary study by PHN CLS investigators, the overall increase in EE rate that was observed during CPG implementation was not sustained, declining from 67% during CPG utilization to 30% in the year following completion of the CLS (4). This outcome was also variable across study sites, with only one site (Site A in both the Bates study and the current study), sustaining the increase in EE in the post CPG follow up period. It is interesting to observe that in the current study, Site A, which showed the highest level of sustainability of CLS EE practices, also had the greatest degree of spillover of EE to other patient groups. It is possible that, had we evaluated for spillover during the 12 month period when the CLS CPG was in use, we would have seen more uniform effects across study sites in spillover of EE, and that differences between study eras across sites were due to site differences in sustainability rather than differences in spillover of CLS practices.

We did not detect any negative consequences from the spillover of CLS extubation practices to patient groups for whom the CPG was not designed. There was no increase in reintubation within 48 hours of extubation, suggesting EE was not inappropriately utilized as a result of CPG spillover. ICU LOS decreased between the Pre CPG and Follow Up era only in the VSD cohort, possibly reflecting the lower complexity and improved postoperative hemodynamic stability of this cohort compared with the others in this study.

There are many factors that could have influenced our findings. The increase in EE observed in the PHN CLS did not translate into shorter ICU or hospital LOS, and as these results were known to the investigators during the follow up period, enthusiasm for EE and the resource utilization that this practice requires may have been dampened. Secular trends in general extubation practices, rather than spillover from the CPG, could have led to increased EE in the Follow Up period, but as there was no change in EE in the 5 control sites across study eras in the PHN CLS, this seems unlikely. However, as collection of extubation data for surgeries not including in PHN CLS at the original control sites was beyond the scope of the present study, we do not know with certainty whether hospitals not exposed to the CLS CPG also saw changes in postoperative extubation practices over the study time period. While we examined across eras many of the patient variables known to impact successful postoperative extubation, some factors, such as cardiopulmonary bypass times and airway anomalies, were not accounted for, and thus we cannot be sure that our results were not impacted by differences in risk factors between the patients in the Pre CPG and Follow Up cohorts. Lastly, use of different data sources, local STS-CHSD for the Pre CPG era and PC4 for the Follow Up period, does raise the possibility of differential classification of outcomes between the two registries. However, given the shared coding by the two datasets and existing literature on the integrity of these two databases for the outcomes under study, data inconsistencies were unlikely to impact our study results. (5, 6, 19).

Conclusions

We observed spillover of early extubation practices from the PHN CLS to the care of infants undergoing lower complexity surgeries not targeted by the CLS CPG, which was sustainable more than one year after completion of the CLS. No changes in extubation practices were seen in infants undergoing higher complexity surgeries. There was no evidence of adverse consequences of spillover to other patient groups. Significant variation in extubation outcomes across study sites suggest different approaches to CPG implementation or other center specific factors may influence the impact of CPGs on clinical practice, and merits further study.