Evaluation of Failure to Rescue as a Quality Metric in Pediatric Heart Surgery: An Analysis of The STS Congenital Heart Surgery Database

Abstract

Background

Failure to rescue (FTR; the probability of death after a complication) has been adopted as a quality metric in adult cardiac surgery, in which it has been shown that high-performing centers with low mortality rates do not have fewer complications, but rather lower mortality in those who experience a complication (lower FTR). It is unknown whether this holds true in pediatric heart surgery. We characterized the relationship between complications, FTR, and mortality in this population.

Methods

Children (0 to 18 years) undergoing heart surgery at centers participating in the Society of Thoracic Surgeons Congenital Heart Surgery Database (2006 to 2009) were included. Outcomes were examined in multivariable analysis adjusting for patient characteristics, surgical risk category, and within-center clustering.

Results

This study included 40,930 patients from 72 centers. Overall in-hospital mortality was 3.7%, 39.3% had a postoperative complication, and the FTR rate (number of deaths in those with a complication) was 9.1%. When hospitals were characterized by in-hospital mortality rate, there was no difference across hospital mortality tertiles in the complication rate in adjusted analysis; however, hospitals in the lowest mortality tertile had significantly lower FTR rates (6.6% versus 12.4%; p < 0.0001). Similar results were seen when evaluating high-severity complications and across surgical risk groups.

Conclusions

This analysis suggests that hospitals with low mortality rates do not have fewer complications after pediatric heart surgery, but instead have lower mortality in those who experience a complication (lower FTR). Further investigation into FTR as a quality metric in pediatric heart surgery is warranted.

Although overall outcomes after congenital heart surgery have improved during the past several decades, recent studies have demonstrated that significant interinstitutional variation in outcome remains for many operations [1]. There are relatively few data regarding the mechanisms underlying this variability. Reducing postoperative complications as a means to improve outcomes has been a general focus in both adults and children undergoing a variety of surgical procedures [2]. However, there is increasing evidence in the adult surgical literature that complication rates tend to be fairly similar across institutions, such that high-performing centers with low mortality rates do not necessarily have a lower rate of complications [3–6]. It has been shown that complications may be related more to patient comorbidities than to hospital characteristics or quality of care [3–6]. In contrast, higher-performing centers appear to have a lower rate of death in those who experience a complication (or lower failure to rescue [FTR] rate). FTR has subsequently been evaluated as a quality metric in adults undergoing a variety of surgical procedures (including adult cardiac surgery), and has been endorsed as a performance measure by the National Quality Forum [3–7]. It has been proposed that initiatives aimed at improving postoperative outcomes across institutions in the adult population should focus on strategies to improve the recognition and management of complications once they occur, in addition to efforts directed toward reducing complications themselves.

There are limited data regarding the relationship among complications, FTR, and center performance in pediatric heart surgery. A previous analysis demonstrated that the rate of postoperative complications was similar across categories of center volume; however, higher-volume centers had a lower rate of death in those with a complication (lower FTR) [8]. The purpose of the present study was to further evaluate FTR as a potential quality metric in pediatric heart surgery. We sought to characterize the relationship of hospital FTR rate and complication rate with mortality rate, as this is the metric of center performance that is currently most widely used. We evaluated this relationship overall, and across different levels of surgical case complexity and type and severity of postoperative complications. An improved understanding of this relationship may have important implications regarding strategies for improving outcomes.

Patients and Methods

Data Source

The Society of Thoracic Surgeons Congenital Heart Surgery (STS-CHS) database currently contains information on more than 200,000 operations performed in North America since 1998 and represents approximately 80% of all US centers performing congenital heart surgery [9]. Data collected include perioperative, operative, and outcomes data on all children undergoing heart surgery at participating centers. Data quality and reliability are evaluated through intrinsic verification of data (eg, identification and correction of missing or out of range values and inconsistencies in values across fields) and a formal process of site visits and data audits [10]. The Duke Clinical Research Institute serves as the data warehouse for the STS databases. This study was approved by the Duke University Institutional Review Board with waiver of consent, and by the STS Access and Publications Committee.

Study Population

This analysis focused on the most recent 4 years of data available in the STS-CHS database at the time of analysis. Between 2006 and 2009, 93 hospitals that submitted data on patients who were 18 years or younger and undergoing a cardiovascular operation (with or without cardiopulmonary bypass) were eligible for inclusion. Hospitals (n = 13) with greater than 15% missing data for key study variables were excluded. Although the STS-CHS database contains nearly complete data for the standard core data fields required to calculate in-hospital mortality, not all centers submit complete data for the other variables in the database such as patient preoperative factors and complications. Therefore, it is standard practice to exclude hospitals with greater than 15% missing data for key study variables to maximize data integrity and minimize missing data [11]. From these 80 hospitals, patients undergoing any operation classified in the Society of Thoracic Surgeons–European Association for Cardio-Thoracic Surgery (STAT) mortality categories were included (category 1 = lowest mortality risk, category 5 = highest mortality risk) [12]. This risk stratification system was recently developed on the basis of empiric data, and is able to classify a greater number of operations compared with other risk stratification systems [12]. Only the first cardiovascular operation of the admission was analyzed. Patients with missing complications or mortality data, as well as intraoperative deaths, were excluded. Finally, hospitals with fewer than 100 cases during the entire 4-year study period (n = 8) were excluded as their limited sample size made comparison with other centers less feasible. Patients from the 72 remaining hospitals included in the final study population were similar to the overall cohort in the STS-CHS database during this period in regard to outcome (in-hospital mortality 3.7% versus 3.9%).

Data Collection

Data collected included patient age, weight, presence of any noncardiac abnormality or genetic syndrome or other preoperative risk factors as defined in the STS-CHS database, and number of prior cardiothoracic surgeries [13]. The primary operation performed was classified according to the STAT mortality category [12]. Center characteristics were also collected, including region of the country and average annual center surgical volume of STAT-classified cases during the study period.

Outcomes

Hospital-level outcomes included in-hospital mortality rate, proportion of patients with any postoperative complication, and the FTR rate. The complications evaluated included 34 postoperative complications collected and defined in the STS-CHS database (version 2.50): cardiovascular (cardiac arrest, mechanical circulatory support, arrhythmia, heart block requiring temporary or permanent pacemaker, low cardiac output, acidosis, pericardial effusion requiring drainage, pulmonary hypertension), pulmonary (pneumothorax, pleural effusion requiring drainage, pneumonia, chylothorax, tracheostomy, phrenic or recurrent laryngeal nerve injury, respiratory insufficiency requiring mechanical ventilation longer than 7 days postoperatively or reintubation), neurologic (transient or permanent neurologic deficit, new-onset seizures), infectious (wound or sternal dehiscence or infection, sepsis, mediastinitis, endocarditis), renal (acute renal failure requiring temporary or permanent dialysis), surgical (unplanned reoperation during the admission, systemic vein obstruction, pulmonary vein obstruction, bleeding requiring reoperation), and other postoperative complications [13]. Of note, delayed sternal closure was not included as a complication for the purposes of this analysis as this is performed electively in many centers and during the study period the database did not distinguish between elective cases and those in which the sternum was left open as a result of hemodynamic compromise.

Analysis

Study variables were described using standard summary statistics. Outcomes were calculated at a hospital level, including in-hospital mortality rate (number of total in-hospital deaths divided by number of total patients), complication rate (number of patients with any postoperative complication divided by number of total of patients), and FTR rate (number of deaths in those with any post-operative complication divided by number of total patients with any post-operative complication). Hospitals were ranked according to adjusted in-hospital mortality and categorized into tertiles, with an equal number of centers in each group. Bayesian hierarchical logistic regression models were used to estimate the hospital-specific in-hospital mortality rates. These models account for the increased variability in outcomes from smaller centers and provide more stable estimates [14]. Specifically, the models provide shrunken estimates toward the population average effect, with the amount of shrinkage inversely correlated to hospital sample sizes. Models were adjusted for the following patient characteristics: age, weight, sex, noncardiac abnormality or genetic syndrome, presence of any other preoperative risk factors (as defined above), any previous cardiothoracic operation, and STAT mortality category.

Separate logistic regression models were then used to evaluate hospital FTR and complication rates across mortality tertiles adjusted for the patient characteristics noted above. The method of generalized estimating equations was used to account for within-center correlation. Odds ratios and 95% confidence intervals are presented. In addition, adjusted complication and FTR rates were calculated for each of the hospital mortality tertiles using the observed to expected (O/E) ratio method. Marginal models for FTR and complication rates adjusted for the patient characteristics noted above were fitted, and the expected numbers of events in each hospital mortality tertile was calculated. The O/E ratio for each tertile, defined as the number of observed events divided by the number of expected events, was then multiplied by the corresponding overall population rate to yield the adjusted FTR and complication rates.

Several sensitivity analyses were also performed. First, a stratified analysis by STAT mortality category was performed to evaluate whether the association of complication rate or FTR rate with hospital mortality varied by surgical risk. For this portion of the analysis, patients were stratified into low-risk (STAT mortality categories 1 through 3) and high-risk (STAT mortality categories 4 and 5) groups, and the analysis was repeated within each subgroup. Next, the potential impact of interinstitutional variation in coding of complications was evaluated by excluding 6 hospitals at the high and low extremes of the distribution of complication rates (n = 12 hospitals total [approximately 15%]) and repeating the analysis in the remaining 60 hospitals. Finally, an analysis was performed to evaluate whether interinstitutional differences in FTR could be explained by differences in the type or severity of complications across hospitals, as it is known that some complications inherently carry greater risk than others. For this portion of the analysis, individual complications were first ranked by their associated mortality risk and stratified into tertiles. The highest-risk complications were then evaluated further. Hospital complication rates for these highest-risk complications and FTR after a high-risk complication (of those with a high-risk complication, the proportion who died) were examined across hospital mortality tertiles using generalized estimating equation models adjusted for the patient characteristics described above.

Results

Study Population Characteristics

A total of 40,930 patients from 72 hospitals were included. Patient characteristics are displayed in Table 1. The included hospitals were geographically diverse (42% South, 24% Midwest, 19% West, 15% Northeast) and had a wide range of average annual volume (median, 194; range, 59 to 833 STAT-classified cases per year).

Table 1.

Study Population Characteristics^a

Variable	n = 40,930
Age	7.0 m (55 days–3.6 y)
≤30 days	8,770 (21%)
>30 days–1 y	15,227 (37%)
1–12 y	13,460 (33%)
>12–18 y	3,475 (9%)
Male	22,521 (55%)
Weight, kg	6.7 (3.9–14.7)
Any STS preoperative risk factor	11,746 (29%)
Any noncardiac/genetic abnormality	12,092 (30%)
≥1 previous cardiothoracic operation	12,520 (31%)
STAT mortality category
1	10,870 (27%)
2	12,548 (31%)
3	6,181 (15%)
4	9,469 (23%)
5	1,862 (4%)

^aData are displayed as median (interquartile range) and number (percent).

STAT = Society of Thoracic Surgeons–European Association for Cardio-Thoracic Surgery; STS = The Society of Thoracic Surgeons.

Outcomes

Overall unadjusted in-hospital mortality in the cohort was 3.7%, 39.3% of patients had any postoperative complication, and the FTR rate (mortality in those with a complication) was 9.1%. Mortality in those without any of the complications coded in the database was rare (0.28%), and overall, 95% of deaths were preceded by at least one of the coded complications.

When hospitals were characterized by adjusted in-hospital mortality rate, there was no difference in the adjusted complication rate across hospital mortality tertiles. However, hospitals in the lowest mortality tertile had significantly lower adjusted FTR rates (6.6% versus 12.4%; p < 0.0001; Table 2; Fig 1). Scatterplots demonstrating correlation between measures (adjusted for the patient characteristics described in the methods) are displayed in Figure 2. There was poor correlation between a hospital’s rank for mortality rate and complication rate, but there was much higher correlation between hospital rank for FTR rate and mortality rate.

Table 2.

Postoperative Complications and Failure to Rescue Across Hospital Mortality Tertiles, Overall and Stratified by The Society of Thoracic Surgeons–European Association for Cardio-Thoracic Surgery Mortality Category^a

Hospital Mortality Tertile	Complications
	Overall	p Value	STAT 1–3	p Value	STAT 4–5	p Value
High	1.16 (0.73–1.87)	0.53	1.14 (0.71–1.84)	0.58	1.23 (0.76–1.99)	0.39
Middle	0.92 (0.49–1.72)	0.80	0.93 (0.50–1.74)	0.82	0.92 (0.49–1.71)	0.78
Low	Reference

Hospital Mortality Tertile	FTR
	Overall	p Value	STAT 1–3	p Value	STAT 4–5	p Value
High	2.20 (1.72–2.82)	<0.0001	1.84 (1.27–2.66)	0.001	2.35 (1.85–2.99)	<0.0001
Middle	1.60 (1.22–2.11)	<0.001	1.53 (1.06–2.21)	0.02	1.61 (1.23–2.11)	<0.001
Low	Reference

^aAdjusted odds ratios and 95% confidence intervals are presented for complications and FTR across hospital mortality tertiles (defined based on adjusted mortality rate), overall and stratified by STAT mortality category.

FTR = failure to rescue; STAT = Society of Thoracic Surgeons–European Association for Cardio-Thoracic Surgery.

Fig 1.

(A) Adjusted complication rate and (B) adjusted failure to rescue rate. Data are adjusted for patient characteristics as outlined in the methods section, and shown for each hospital mortality tertile (defined based on adjusted mortality rate).

Fig 2.

Correlation between measures. Scatterplots demonstrate the relationship between hospital rank for mortality rate versus (A) complication rate, and hospital rank for mortality rate versus (B) failure to rescue (FTR) rate. All measures are adjusted for the patient characteristics outlined in the methods.

Sensitivity Analyses

When the analysis was stratified by STAT mortality category, results across levels of surgical risk were similar to the main results. There was no difference in the adjusted complication rate across hospital mortality tertiles in either the low- (STAT mortality categories 1 through 3) or high-(STAT mortality categories 4 and 5) risk categories, although hospitals with lower mortality had lower FTR in both of the surgical risk groups (Table 2). Next, hospitals at the upper and lower extremes of the distribution of complication rates were excluded and the analysis repeated. These results were also similar to the main results. There remained a significant association of hospital mortality with the FTR rate, but not with complication rate (Fig 3). Finally, the potential impact of complication severity on our findings was investigated. Complications in the highest-risk group (along with their associated in-hospital mortality risk) included acute renal failure requiring permanent dialysis (84%), acute renal failure requiring temporary dialysis (58%), mechanical circulatory support (57%), pulmonary vein obstruction (53%), cardiac arrest (53%), neurologic deficit persisting at discharge (32%), systemic vein obstruction (32%), low cardiac output (29%), bleeding requiring reoperation (29%), acidosis (28%), and sepsis (27%). There was not a significant difference in the proportion of patients with these high-risk complications across hospital mortality tertiles (Table 3). Similar to the main results, hospitals with lower mortality had significantly lower FTR after a high-risk complication (Table 3).

Fig 3.

Sensitivity analysis showing the relationship of hospital mortality with complications and failure to rescue excluding centers at extremes of complication coding. Adjusted odds ratios (black boxes), and 95% confidence intervals (lines) comparing (A) complications and (B) failure to rescue across hospital mortality tertiles are presented for the overall cohort and for the subgroup of hospitals remaining after those at the extremes of the distribution of complication coding were excluded as described in the methods. In both cohorts, there is no significant relationship between hospital mortality and complications; however, there is a significant association of failure to rescue with hospital mortality.

Table 3.

High-Risk Complications and Failure to Rescue After a High-Risk Complication, According to Hospital Mortality Tertiles^a

High-Risk Complications
Hospital Mortality Tertile	Adjusted Odds Ratio (95% CI)	p Value
High	1.55 (0.89–2.70)	0.12
Middle	1.24 (0.67–2.29)	0.49
Low	Reference

FTR After a High-Risk Complication
Hospital Mortality Tertile	Adjusted Odds Ratio (95% CI)	p Value
High	1.99 (1.31–3.02)	0.001
Middle	1.25 (0.82–1.92)	0.30
Low	Reference

^aAdjusted odds ratios and 95% confidence intervals are presented for high-risk complications and FTR after a high-risk complication across hospital mortality tertiles (defined based on adjusted mortality rate).

CI = confidence interval; FTR = failure to rescue.

Comment

These analyses suggest that high-performing pediatric heart centers with lower mortality rates do not necessarily have fewer complications, but instead have lower mortality in those who experience a complication (or lower FTR). These data may have important health policy implications, as they suggest that initiatives aimed at improving outcomes may need to focus on improving recognition and management of complications once they occur, in addition to reducing the occurrence of complications themselves. The results of our analyses are consistent with studies in the adult population in which FTR has been studied more extensively [3–6]. Although it seems intuitive that higher-performing hospitals would have lower rates of postoperative complications, numerous studies in adults have shown that complication rate and hospital performance are poorly correlated [3–6]. Further, it has been shown that complications are largely related to patient comorbidities or preoperative risk factors (and may be unavoidable in many cases), as differences across centers in complication rates are no longer evident after risk adjustment [6, 8].

In contrast, it has been shown that the occurrence of a postoperative complication, even seemingly minor complications, can significantly alter a patient’s subsequent risk of death, and that the cohort of patients who experience a complication represents a high-risk group [15]. Silber and colleagues [4, 6, 15] first coined the term failure to rescue, and demonstrated the correlation between FTR and hospital performance. However, initial studies of FTR were met with certain criticisms. Many of these analyses used administrative data sources, which can be limited by difficulties in identifying complications, and other issues related to miscoding. However, subsequent studies using prospectively collected clinical data validated the relationship of FTR with hospital performance [3]. Ghaferi and colleagues [3] studied 84,730 patients who had undergone inpatient general or vascular surgery from 2005 to 2007 using data from the American College of Surgeons National Surgical Quality Improvement Program. They found similar rates of postoperative complications across centers, including those with very high or very low mortality rates. However the FTR rate was much lower in high-performing versus low-performing hospitals (6.8% versus 16.7%) [3]. The FTR rate has been endorsed as a performance measure by the National Quality Forum [7]. Subsequent studies of FTR in adults have further investigated center-level structure and process measures that may be related to FTR rates, and have evaluated in more detail specific complications, including the type and time course of multiple complications in patients who subsequently died [4, 7, 16–18]. Studies such as these are needed to further investigate FTR as a metric in the pediatric population.

This evaluation of FTR in the pediatric heart surgery population demonstrates that the complications currently captured in the STS-CHS database appear to be adequate for creation of an FTR measure. Our data showed that 95% of deaths were preceded by a complication collected in the database, which is the recommended proportion for constructing an FTR metric in previous adult studies [4]. Our overall findings are similar to those from previous adult studies in that the FTR rate, but not the complication rate, appears to correlate with hospital performance whether evaluated using the surrogate of center volume or assessed by the adjusted mortality rate as in the present study [8]. Our data also suggest that the relationship of complications and FTR with hospital mortality is consistent across different levels of surgical case complexity, holds true when centers at the extremes of complication coding are excluded, remains evident when only the subset of high-risk complications is considered, and is not related to significant differences in the distribution or severity of complications across institutions.

There are several potential advantages to the use of FTR as a quality measure in addition to other metrics. First, FTR may be less sensitive to differences in patient characteristics or comorbidities, particularly compared with the evaluation of complications alone [4, 6, 8]. Second, it has been suggested that FTR may have greater power in certain cases to compare outcomes across hospitals compared with mortality [4]. This will require further investigation in the pediatric population. Finally, FTR may provide a potential mechanism underlying differences in outcome across centers. Investigation into factors impacting a center’s ability to recognize and respond to complications may identify targets for quality-improvement initiatives aimed at reducing FTR and improving outcomes [18].

Limitations

The limitations of this study are primarily related to the nature of the STS-CHS database. The database does not capture every possible postoperative complication. Nonetheless, we did find that 95% of deaths were preceded by at least one of the complications collected in the database, which is the recommended proportion for constructing an FTR metric in previous adult studies [4]. In addition, death in those without any complication coded in the database was extremely rare (0.28%), supporting the clinical validity of the data set. Although there are uniform definitions for the complications collected in the database, there may still be variability in the coding of complications across centers. However, our results did not appear to be sensitive to the inclusion of centers at the extremes of complication coding, as exclusion of these centers yielded similar results to the main analysis. Not all centers submit complete data and not all US centers participate in the database currently. However, the similarity of mortality outcomes in our study cohort to that of the overall cohort of STS centers during this period supports the generalizability of our results. We were not able to evaluate the relationship of center-level variables such as hospital structure or processes, training and availability of personnel, or nursing variables on outcomes as these variables are not currently collected in the database. However, the database will soon begin collecting center-level variables in a quality module, which may be related to outcomes, including FTR, in subsequent analyses. In addition, we were not able assess the time course of events in patients with multiple postoperative complications as timestamp data are currently not collected in the database, although this may be added at a future date. Because of this we are not able to determine the sentinel (or first) complication, or the final cause of death in this study. Finally, in this analysis we evaluated the relationship of FTR and complications with hospital mortality, as this is the most widely used metric of center performance currently. However, it should be noted that there can be limitations to the use of mortality alone to evaluate hospital performance, and it may be that a combination of metrics, including some of those evaluated in this study, is more reliable [19, 20].

Conclusions

This study suggests that the higher mortality rate observed at poorer-performing hospitals may in part be related to a higher rate of mortality in those with postoperative complications, rather than a higher rate of postoperative complications themselves. Further characterization of variation in the FTR rate across centers, time course and relationship between multiple postoperative complications and mortality, and association of FTR with factors affecting a hospital’s ability to respond to and manage complications is needed, and these data may lead to strategies to improve quality of care and outcomes in children undergoing heart surgery.