Abstract
Background
Risk‐adjusted adverse event (AE) rates have been used to measure the quality of pediatric and congenital cardiac catheterization laboratories. In other settings, failure to rescue (FTR) has demonstrated utility as a quality metric.
Methods and Results
A multicenter retrospective cohort study was performed using data from the IMPACT (Improving Adult and Congenital Treatment) Registry between January 2010 and December 2016. A modified FTR metric was developed for pediatric and congenital cardiac catheterization laboratories and then compared with pooled AEs. The associations between patient‐ and hospital‐level factors and outcomes were evaluated using hierarchical logistic regression models. Hospital risk standardized ratios were then calculated. Rankings of risk standardized ratios for each outcome were compared to determine whether AEs and FTR identified the same high‐ and low‐performing centers. During the study period, 77 580 catheterizations were performed at 91 hospitals. Higher annual hospital catheterization volume was associated with lower odds of FTR (odds ratio: 0.68 per 300 cases; P=0.0003). No association was seen between catheterization volume and odds of AEs. Odds of AEs were instead associated with patient‐ and procedure‐level factors. There was no correlation between risk standardized ratio ranks for FTR and pooled AEs (P=0.46). Hospital ranks by catheterization volume and FTR were associated (r=−0.28, P=0.01) with the largest volume hospitals having the lowest risk of FTR.
Conclusions
In contrast to AEs, FTR was not strongly associated with patient‐ and procedure‐level factors and was significantly associated with pediatric and congenital cardiac catheterization laboratory volume. Hospital rankings based on FTR and AEs were not significantly correlated. We conclude that FTR is a complementary measure of catheterization laboratory quality and should be included in future research and quality‐improvement projects.
Keywords: health services research, outcomes research, pediatrics
Subject Categories: Congenital Heart Disease, Health Services, Quality and Outcomes
Clinical Perspective
What Is New?
A failure to rescue (FTR) metric was developed specifically for the pediatric and congenital catheterization laboratory setting and compared with pooled adverse events (AEs).
Risk of pooled AEs was related to participant and procedural factors, whereas FTR risk was associated with hospital volume.
Hospital ranks using risk‐adjusted AEs and FTR differed significantly.
What Are the Clinical Implications?
FTR as a quality metric adds unique information about programmatic quality.
FTR should be included in future research studies, quality‐improvement efforts, and evaluations of programmatic performance.
Cardiac catheterization remains the gold standard for hemodynamic evaluation of young patients with cardiac disease and as a means of intervening for an increasing range of conditions. Adverse events (AEs) in the pediatric and congenital cardiac catheterization laboratory (PCCL) are relatively uncommon. Nonetheless, PCCL procedures are a source of significant morbidity.1, 2 Previous efforts to define quality metrics have pooled different AEs.3, 4, 5, 6, 7, 8 Pooled AEs were chosen over mortality because deaths following catheterization are rare, and concerns exist regarding (1) attributability of deaths to the catheterization procedure, (2) significant medical comorbidity, and (3) frequent crossover to and from the care of cardiac surgeons.
An important question is whether the risk‐adjusted AE rate is the single, optimal quality measure for PCCL programs. Failure to rescue (FTR), defined as death following a major AE, was introduced as an alternative to the pooled AE outcome metric.9 In diverse settings, FTR has demonstrated attractive characteristics relative to both pooled AE and mortality rates. In these settings, significant associations have been demonstrated between FTR and known measures of hospital quality, whereas the factors associated with pooled AEs have been patient‐level factors.9, 10, 11 In addition, ranks of hospitals based on risk‐adjusted AEs differ significantly from those based on FTR.11 These findings suggest that FTR has value in measuring the quality of care delivered at different hospitals. Although initially developed in a cohort of adult surgical patients,9 FTR has been applied in a broad array of medical and surgical contexts.9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23
Using data from the IMPACT (Improving Adult and Congenital Treatment) Registry, we performed a multicenter retrospective cohort study to develop an FTR metric adapted to the PCCL environment and to determine whether that metric provided value in addition to that provided by risk‐adjusted AEs. We sought to determine what patient‐, procedure‐, and hospital‐level factors were associated with risk of complications and FTR and whether hospital rankings based on FTR and pooled AEs were correlated. We hypothesized (1) that patient and procedure factors would be associated with the odds of pooled AEs but not FTR and that, conversely, potential markers of programmatic quality (specifically, hospital procedural volume) would be associated with the odds of FTR but not pooled AEs and (2) that rankings of hospitals by FTR and AEs would not be significantly correlated. Together, these findings would show that FTR provided unique information in addition to that provided by risk‐adjusted AEs.
Methods
Data Source
IMPACT is a clinical registry funded by the American College of Cardiology and managed by the National Cardiovascular Data Registry with data from 91 pediatric and general hospitals performing cardiac catheterizations in children and adults with congenital heart disease at the time of this analysis. Participating centers collect demographics, medical and surgical history, procedural information, and AEs through hospital discharge on all patients undergoing cardiac catheterization. Data are recorded using standardized data elements and definitions. The database is subject to quality assurance standards.24 The current study used data from IMPACT v1.0.1. The institutional review board of the Children's Hospital of Philadelphia reviewed the proposed project and determined that it did not represent human subjects research in accordance with the Common Rule (45 CFR 46.102(f)). Data, analytic methods, and study materials will not be made available to other researchers for the purposes of reproducing the results or replicating the procedures because that would not be consistent with the data‐use agreement between the study staff and the American College of Cardiology Foundation and National Cardiovascular Data Registry.
Study Population
Cases performed between January 2010 and December 2016 at centers contributing to IMPACT in patients between birth and 25 years of age were eligible for inclusion. Cases reported as emergent or salvage status were excluded because the patients were already in jeopardy; therefore, these cases would not be informative about the quality of care in the catheterization laboratory. For the same reasons, cases involving patients who were already receiving mechanical circulatory (extracorporeal membrane oxygenation or ventricular assist devices) were also excluded. Although patients with recent cardiac surgery were potentially at higher risk of adverse outcomes, their cases represent a common and important part of clinical practice and so were not excluded unless urgency of the procedure was classified as emergency or salvage. As noted below, statistical adjustment for history of recent cardiac surgery was performed to mitigate any bias introduced by differential utilization of catheterization in the postoperative period between hospitals.
Study Data
Data were extracted directly from the IMPACT Registry, specifically demographics, medical history (including preprocedure risk factors), hemodynamic data, and details of the procedure (specific interventions, anesthesia at start of the case, device used, and presence of a trainee). AEs recorded in IMPACT (in‐hospital mortality; cardiac arrest; new arrhythmia; new heart valve regurgitation; tamponade; air embolus; embolic stroke; device malposition; device embolization; airway event; blood transfusion; initiation of dialysis; new extracorporeal membrane oxygenation; new ventricular assist device; unplanned cardiac, vascular, or other surgery; vascular complication requiring treatment; repeat catheterization [all before discharge], and other events) were recorded. Of note, the registry instructs centers to report only unplanned surgical and catheterization procedures that are attributable to a catheterization complication. Hospital procedural volume was calculated as prorated annual averages, as described previously.25, 26, 27
Definitions
FTR was originally defined as death following a complication, reflecting the hospital's inability to prevent that progression.9 We chose to adapt this definition to the PCCL environment with 2 changes. First, we counted all deaths occurring within 2 days from catheterization. In previous FTR studies of surgical patients, it has been assumed that all deaths within 30 days were attributable to preceding procedure‐related complications.9, 12 In the PCCL, patients may undergo technically successful catheterizations without complication and then subsequently die during the same hospitalization either after cardiac surgery or because of progression of their condition. In a multicenter series, only 10% of deaths within 30 days of PCCL were attributable (after adjudication) to the catheterization.28 We reviewed the incident and cumulative rate of deaths after catheterization and found that the 2‐day mark was a natural qualitative cut point in the incidence of postcatheterization deaths. This approach was also consistent with previous research.25, 26 These choices were made to minimize inclusion of deaths due to other causes and were necessary because it is not possible to review and adjudicate all deaths and other events in IMPACT. Deaths and other events that occurred in the same hospitalization but after an elective surgical procedure were not included regardless of timing. This definition of deaths was used for both FTR and pooled AEs. Second, because we intended to measure the quality of catheterization laboratory care, we included other catastrophic AEs for which care escalated beyond the capacity of the catheterization laboratory team in our list of failures (Table 1). We acknowledge that in some cases, mechanical circulatory support was initiated electively or semielectively in catheterization patients. For our analysis, these failure events were also considered separately as catastrophic AE.
Table 1.
Proximal events |
New arrhythmia |
New heart valve regurgitation |
Cardiac tamponade |
Air embolus |
Embolic stroke |
Device malposition |
Device embolization |
Airway event |
Initiation of dialysis |
Unplanned cardiac, vascular, or other surgery (due to catheterization complication)a |
Vascular complication requiring treatment (due to catheterization complication)a |
Repeated catheterization (due to catheterization complication)a |
Other major events |
Catastrophic events |
Death within 2 days of catheterization |
Cardiac arrest |
Initiation of mechanical circulatory support |
Initiation of extracorporeal membrane oxygenation |
Unplanned cardiac, vascular, or other surgery (due to catheterization complication)a |
Vascular complication requiring treatment (due to catheterization complication)a |
Repeat catheterization (due to catheterization complication)a |
Outcomes were determined using IMPACT (Improving Adult and Congenital Treatment) Registry standard definitions. Failure to rescue was defined as a catastrophic adverse outcome occurring in a case with a proximal event. Pooled adverse events encompass any of the proximal or catastrophic events.
These events are defined as being due to a catheterization complication, so they were included in both proximal and catastrophic events.
Next we evaluated the possible AEs after catheterization and created a list of those that could potentially lead to a catastrophic adverse outcome (Table 1). Blood transfusion was the most common AE noted in the registry, but it was not included as a proximal event because it was judged not likely to be directly and independently responsible for a catastrophic event. This choice was supported by the observation that removing it from the list of preceding events did not change the precedence rate (see below). Of note, repeated catheterization and surgeries are recorded in IMPACT only when the procedure is attributable to a catheterization complication. Consequently, even if no other AE were recorded, cases in which unplanned surgeries (cardiac, vascular, or other) or repeated catheterizations were reported were recorded as having both proximal and catastrophic AEs. The proportion of these unplanned procedures with proximal events is reported in Table S1.
To evaluate whether the list of proximal events was comprehensive, we calculated the proportion of catastrophic AEs with a preceding proximal event (the precedence rate). In this sample, the precedence rate was 70% (687/977). This rate is close to what has been seen in other studies using FTR.22 It may also reflect that patients arriving in the PCCL are potentially more medically complicated (with greater possibility of deterioration independent of a preceding complication) than older patients undergoing isolated surgical procedures.
To summarize, the definition of our modified FTR metric is the occurrence of one of the catastrophic AEs after one of the proximal events.
Statistical Analysis
Descriptive statistics were calculated. Observed rates for total AEs, proximal AEs, catastrophic AEs, and FTR were calculated.
The goals of subsequent analyses were to compare the characteristics of the different study outcomes as quality metrics. First, generalized linear models with binomial frequency distribution, adjusted for the listed covariates as fixed effects and clustered by hospital (using a random intercept for each hospital, implemented using glimmix in SAS v9.4; SAS Institute), were calculated to evaluate the degree to which patient, procedure, and hospital factors were associated with each outcome of interest. Additional random effects (eg, random slope) and interaction were not explored. FTR was the primary outcome of interest. Pooled AEs (including and excluding deaths), proximal AEs, and catastrophic AEs were secondary outcomes. We hypothesized that odds of pooled AEs would be associated with patient and procedural factors and that odds of FTR would not be associated with these factors. Moreover, we sought to determine whether markers of hospital quality were associated with risk of these outcomes, hypothesizing that odds of FTR would be associated with these factors and that odds of pooled AEs would not. To our knowledge, no hospital characteristics are unequivocally associated with the quality of a PCCL program. Hospital procedural volume has been associated with improved outcomes across disparate medical fields.29 Consequently, hospital annual catheterization volume was used as the primary hospital characteristic of interest.
Covariates were identified before model calculation and included patient, procedure, and hospital characteristics. Patient characteristics were age by category, sex, race, previous cardiac surgery in the prior 30 days, cardiac catheterization in the prior 30 days, preprocedure inotropes, preprocedure vasodilator, preprocedure sepsis, chronic lung disease, genetic syndrome, renal insufficiency, hemodynamic vulnerability (as defined previously7), low systemic arterial saturation, low mixed venous saturation, high mean pulmonary pressure, and elevated systemic ventricular end‐diastolic pressure. Procedure characteristics were procedural risk category as defined in the Catheterization for Congenital Heart Disease Adjustment for Risk Method,4, 6, 7 use of general anesthesia versus conscious sedation, and presence of a trainee. Hospital characteristics were procedural volume >35% adult and academic or university hospital. These separate covariates were used rather than the IMPACT risk‐adjustment model30 because that risk model combines procedures and patient ages. It was also not clear, a priori, that this component of the risk model would be equally applicable to all outcomes, and it was not desirable to forego information about both ages and procedure types. Hemodynamic vulnerabilities were considered individually rather than as a number of vulnerabilities, as had been done previously,7 to avoid bias from missing data and to evaluate whether the effect of hemodynamic vulnerabilities differed for AEs and FTR. To determine whether either deviation from the validated risk adjustment model introduced bias, sensitivity analyses were performed using the most recent procedure codes and the sum of the number of hemodynamic vulnerabilities present. No changes in the observed results were seen (data not shown). A third sensitivity analysis was performed by recalculating all models excluding patients with cardiac surgery in the prior 30 days. Again, no changes in the reported results were seen (data not shown). The threshold of >35% cases in adults was chosen based on examination of the distribution of adult cases in the hospitals contributing data to the IMPACT Registry, which identified a qualitatively different cluster of hospitals, as has been reported previously.31, 32
The primary model used FTR as an outcome. Secondary analyses planned before the beginning of the analyses included calculating analogous models for the risk of (1) all AEs, (2) catastrophic AEs, (3) all AEs excluding death, and (4) proximal AEs. It was also recognized that the database includes multiple catheterization procedures in individual subjects. To address this, a sensitivity analysis restricted to the first catheterization of each participant was performed for our primary model, as in previous studies.25, 26 To ensure that the associations between hospital factors and outcomes were not due to correlation between these factors, additional models were calculated with only 1 of the 3 center factors for each outcome, confirming that no change in the observed associations was seen.
Next we sought to evaluate whether hospital rankings based on case‐mix–adjusted AEs and FTR identified the same high‐ and low‐performing hospitals. For each hospital, a risk standardized ratio (RSR) was calculated, and hospitals were ranked. RSR is a ratio of the predicted outcome for each hospital to the expected rate for a hospital with similar case mix based on previously calculated hierarchical models. Correlation between rankings of the RSR for FTR, composite AEs, and catastrophic AEs were evaluated using Spearman rank correlation coefficients, as has been done in previous studies using FTR.10, 22 Similarly, the association between the aforementioned outcomes and hospital PCCL volume was also evaluated.
Not all hemodynamic factors were recorded in every case. To avoid the potential bias introduced by excluding cases with missing hemodynamics, a variable for missing was included, as described previously.27, 33 Multiple imputation was not used because the missing data were not plausibly predicted by collected data. As described, the primary analysis and secondary analyses (exposure, outcome, and covariates) were defined before analysis. To avoid bias, no post hoc model refinement was performed. No formal steps were taken to penalize for multiple comparisons, and analyses beyond the primary analyses should be considered exploratory. A threshold for statistical significance was set at P<0.05. All data analysis was performed using SAS 9.4.
Results
Study Population
Over the study period, 77 580 procedures were performed on 53 056 individual participants at 91 hospitals (Figure 1). Of the cases included, the median age of participants was 4 years (interquartile range: 0–12 years). Overall, 47% were female, and 70% were white. Patient and procedural characteristics are summarized in Table 2. Of the hospitals contributing data to IMPACT, 46% (42/91) were university or teaching hospitals, and 10% (9/91) had a large proportion of their reported catheterizations in adults (>35%). The median annual catheterization volume of hospitals included in the study was 284 cases (interquartile range: 117–416 cases).
Table 2.
Characteristic | Result |
---|---|
No. of cases | 77 580 |
No. of participants | 53 056 |
No. of hospitals | 91 |
Age, y, median (IQR) | 4.0 (0–12) |
Age group | |
<30 d | 5 (4010) |
>30 d to 1 y | 21 (16 392) |
1–8 y | 38 (29 104) |
8–18 y | 28 (21 894) |
>18 y | 8 (6180) |
Female sex | 47 (36 108) |
Race | |
White | 70 (54 648) |
Black | 17 (13 426) |
Other | 12 (9506) |
Cardiothoracic surgery within 30 d prior | 2 (1568) |
Cardiac catheterization within 30 d prior | 4 (3112) |
Chronic lung disease | 6.5 (5009) |
Single ventricle | 21 (16 018) |
Genetic syndrome | 12 (8912) |
Renal insufficiency | 2.8(2166) |
Preprocedure inotropes | 6.8 (5273) |
Preprocedure vasodilator | 9.6 (7456) |
Hemodynamic vulnerability | |
Systemic arterial saturation | 27 (19 542/71 500) |
Mixed venous saturation | 13 (8981/68 853) |
Pulmonary artery pressure | 18 (11 184/62 658) |
Systemic ventricular end‐diastolic pressure | 6 (3158/50 575) |
Procedure risk category | |
1 | 41 (31 966) |
2 | 33 (25 586) |
3 | 19 (14 625) |
4 | 7 (5381) |
Sedation strategy | |
General anesthesia | 87 (67 564) |
Conscious sedation | 12 (9144) |
Trainee present | 63 (48 743) |
Case performed at hospital with >35% adult catheterization volume | 2 (1300) |
Case performed at a teaching institution | 93 (71 594) |
Data are shown as % (n) except as noted. IQR indicates interquartile range.
Outcomes
AEs occurred in 4.7% of cases overall (Table 3). The risk of the composite outcome for catastrophic AEs was 1.2%. The risk of proximal AEs was 4.4% (frequencies of individual proximal AEs are summarized in Table S2). The risk of catastrophic AEs after proximal AEs (observed FTR) was 20.3%.
Table 3.
Outcome | Result, % (n) |
---|---|
Any AE | 4.7 (3682) |
Catastrophic adverse outcome | 1.2 (960) |
Death within 2 d | 0.2 (141) |
Cardiac arrest | 0.6 (465) |
Initiation of ECMO/VAD | 0.2 (138) |
Unplanned cardiac operation due to catheterization complication | 0.3 (232) |
Unplanned vascular operation due to catheterization complication | 0.1 (41) |
Unplanned other operation due to catheterization complication | 0.2 (153) |
Unplanned catheterization due to catheterization complication | 0.3 (252) |
Proximal adverse outcomes | 4.4 (3416) |
FTR | 20.3 (694/3416) |
AE indicates adverse event; ECMO, extracorporeal membrane oxygenation; FTR, failure to rescue; VAD, ventricular assist device.
Factors Associated With FTR, AE, and Catastrophic AE Rates
Hierarchical models were calculated for each major outcome (Table 4). These models demonstrated that the adjusted risk of FTR was significantly lower at higher volume hospitals (odds ratio [OR] per 300 cases: 0.68; 95% CI, 0.55–0.84; P=0.0003). The risk of FTR was lower at hospitals with a high proportion of cases in adults (OR: 0.16; 95% CI, 0.05–0.55; P=0.003). Preprocedural receipt of inotropes (OR: 3.05; 95% CI, 2.43–3.83; P<0.0001), renal insufficiency (OR: 2.42; 95% CI, 1.54–3.79; P=0.0001), and low mixed venous saturation (OR: 1.39; 95% CI, 1.09–1.79; P=0.009) were positively associated with FTR, whereas surgery in the prior 30 days (OR: 0.16; 95% CI, 0.1–0.25; P<0.0001) was negatively associated with FTR. Procedure risk categories 2 and 3 were negatively associated with FTR relative to risk category 1 (OR: 0.70; 95% CI, 0.51–0.95; P=0.02; and OR: 0.72; 95% CI, 0.52–0.99; P=0.04, respectively).
Table 4.
FTR | All AEs | Catastrophic AEs | ||||
---|---|---|---|---|---|---|
OR (95% CI) | P Value | OR (95% CI) | P Value | OR (95% CI) | P Value | |
Age group | ||||||
<30 d | 0.82 (0.56–1.18) | 0.29 | 2.62 (2.27–3.03) | <0.0001 | 2.25 (1.71–2.96) | <0.0001 |
30 d to 1 y | 1.21 (0.90–1.62) | 0.21 | 1.55 (1.39–1.73) | <0.0001 | 1.92 (1.55–2.39) | <0.0001 |
1–8 y | 1 (NA) | NA | 1 (NA) | NA | 1 (NA) | NA |
8–18 y | 1.29 (0.97–1.73) | 0.08 | 0.99 (0.89–1.10) | 0.86 | 1.08 (0.87–1.34) | 0.51 |
≥18 y | 1.28 (0.80–2.06) | 0.31 | 0.91 (0.76–1.09) | 0.31 | 1.04 (0.73–1.48) | 0.82 |
Male sex | 0.89 (0.74–1.07) | 0.20 | 1.05 (0.98–1.13) | 0.14 | 0.96 (0.84–1.09) | 0.52 |
Race | ||||||
White | 1 (NA) | NA | 1 (NA) | NA | 1 (NA) | NA |
Black | 1.04 (0.81–1.35) | 0.75 | 1.02 (0.98–1.13) | 0.14 | 1.06 (0.88–1.26) | 0.56 |
Other | 1.21 (0.9–1.62) | 0.22 | 1.00 (0.89–1.12) | 0.95 | 1.21 (0.98–1.49) | 0.08 |
Surgery (previous 30 d) | 0.16 (0.1–0.25) | <0.0001 | 10.15 (8.80–11.69) | <0.0001 | 1.05 (0.70–1.56) | 0.82 |
Catheterization (previous 30 d) | 1.24 (0.71–2.15) | 0.45 | 0.65 (0.53–0.80) | <0.0001 | 1.00 (0.69–1.46) | 0.98 |
Preprocedure medications | ||||||
Inotrope | 3.05 (2.43–3.83) | <0.0001 | 2.37 (2.14–2.62) | <0.0001 | 4.94 (4.20–5.82) | <0.0001 |
Vasodilators | 0.96 (0.72–1.27) | 0.77 | 1.02 (0.91–1.14) | 0.79 | 1.08 (0.89–1.31) | 0.45 |
Preprocedure conditions | ||||||
Sepsis | 2.19 (0.94–5.07) | 0.07 | 1.16 (0.78–1.72) | 0.47 | 1.62 (0.94–2.79) | 0.08 |
Chronic lung disease | 0.63 (0.43–0.90) | 0.01 | 0.98 (0.86–1.12) | 0.79 | 0.8 (0.61–1.03) | 0.09 |
Genetic syndrome | 0.76 (0.58–0.99) | 0.05 | 1.22 (1.11–1.35) | <0.0001 | 1.01 (0.83–1.23) | 0.91 |
Renal insufficiency | 2.42 (1.54–3.79) | 0.0001 | 1.28 (1.05–1.58) | 0.02 | 1.98 (1.47–2.68) | <0.0001 |
Systemic arterial saturation | ||||||
Low vs normal | 0.95 (0.77–1.19) | 0.68 | 1.14 (1.05–1.24) | 0.003 | 1.08 (0.92–1.27) | 0.36 |
Low vs missing | 1.17 (0.8–1.7) | 0.42 | 0.97 (0.83–1.13) | 0.69 | 1.07 (0.82–1.40) | 0.63 |
Mixed venous saturation | ||||||
Low vs normal | 1.39 (1.09–1.79) | 0.009 | 1.53 (1.38–1.69) | <0.0001 | 1.85 (1.54–2.21) | <0.0001 |
Low vs missing | 1.22 (0.88–1.70) | 0.22 | 1.20 (1.05–1.38) | 0.006 | 1.39 (1.10–1.77) | 0.007 |
Pulmonary artery pressure | ||||||
High vs normal | 1.00 (0.78–1.29) | 1.00 | 1.41 (1.28–1.56) | <0.0001 | 1.56 (1.30–1.86) | <0.0001 |
High vs missing | 1.11 (0.85–1.43) | 0.44 | 1.35 (1.22–1.49) | <0.0001 | 1.53 (1.27–1.84) | <0.0001 |
Systemic ventricular end‐diastolic pressure | ||||||
High vs normal | 0.86 (0.57–1.29) | 0.46 | 1.45 (1.23–1.69) | <0.0001 | 1.43 (1.09–1.88) | 0.009 |
High vs missing | 1.15 (0.93–1.43) | 0.21 | 1.03 (0.94–1.11) | 0.56 | 1.13 (0.97–1.32) | 0.13 |
Procedure group | ||||||
1 | 1 (NA) | NA | 1 (NA) | NA | 1 (NA) | NA |
2 | 0.70 (0.51–0.95) | 0.02 | 1.54 (1.37–1.73) | <0.0001 | 1.03 (0.81–1.30) | 0.81 |
3 | 0.72 (0.52–0.99) | 0.04 | 1.70 (1.51–1.92) | <0.0001 | 1.14 (0.9–1.45) | 0.28 |
4 | 0.89 (0.62–1.29) | 0.55 | 2.05 (1.78–2.36) | <0.0001 | 1.49 (1.14–1.96) | 0.004 |
Trainee present | 0.96 (0.74–1.23) | 0.72 | 1.04 (0.93–1.16) | 0.52 | 1.01 (0.84–1.22) | 0.89 |
General anesthesia vs IV sedation | 1.17 (0.81–1.7) | 0.40 | 1.28 (1.12–1.46) | 0.0004 | 1.41 (1.09–1.84) | 0.01 |
Hospital characteristics | ||||||
Adult cases >35% of total | 0.16 (0.05–0.55) | 0.003 | 1.37 (0.82–2.28) | 0.23 | 0.73 (0.34–1.57) | 0.42 |
University hospital | 1.01 (0.72–1.41) | 0.95 | 1.22 (0.97–1.52) | 0.09 | 1.16 (0.87–1.53) | 0.31 |
Annual catheterization volume (per 300 cases) | 0.68 (0.55–0.84) | 0.0003 | 0.95 (0.83–1.10) | 0.53 | 0.79 (0.66–0.94) | 0.009 |
AE indicates adverse event; FTR, failure to rescue; IV, intravenous; NA, not available; OR, odds ratio.
In contrast, composite AEs were not significantly associated with procedural volume (P=0.53) but were significantly associated with patient‐level factors (age, preprocedure inotropes, recent surgery or catheterization, genetic syndrome, renal insufficiency, and previously identified markers of hemodynamic vulnerability) and procedure‐level factors (procedure group and use of general anesthesia).
A similar pattern of positive association was seen between many of the previously identified patient‐level factors (age, inotropes, renal insufficiency, 3 of the 4 markers of hemodynamic vulnerability) and procedure‐level factors (procedure group 4 and receipt of general anesthesia) and catastrophic adverse outcome. Increasing PCCL volume was associated with reduced odds of catastrophic adverse outcome (OR: 0.79; 95% CI, 066–0.94; P=0.009).
Analogous models were calculated for AEs with deaths excluded and proximal AEs (Table S3), with both models resembling the model for all AEs. A sensitivity analysis was performed for the FTR model restricted to the first catheterization in the sample with no significant changes in the point estimates (Table S4).
To evaluate interactions between hospital factors, models including only procedural volume, adult volume >35%, and teaching hospital versus non–teaching hospital were calculated and demonstrated no differences from the primary model (Tables S5–S7).
Ranking Programs
For each hospital, standardized ratios of observed to expected events were calculated for FTR, all AEs, and catastrophic AEs. There was no correlation between hospital rankings by RSR for all AEs and RSR for FTR (Spearman r=−0.08, P=0.46; if deaths are excluded, Spearman r=−0.14, P=0.23). In contrast, hospital ranking based on RSR for FTR and RSR for catastrophic AEs were strongly correlated (Spearman r=0.65, P<0.05; Figure 2).
Rank correlations based on hospital catheterization volume and our outcomes were also performed. There was a significant association between the ranks of hospitals based on procedural volume and RSR for FTR (r=−0.28, P=0.01; Figure 3) but not between annual catheterization volume and RSR for all AEs (P=0.32). There was of note a significant association between procedural volume rank and RSR for catastrophic volume (r=−0.21, P=0.048; data not shown).
Discussion
In this retrospective multi‐center cohort study, we sought to evaluate a modified version of FTR as a potential quality metric for PCCL programs. We compared its properties with a composite outcome of pooled major AEs that has been the standard in previous risk‐adjustment models.6, 7, 27 As in previous studies, the odds of pooled AEs were associated with a number of patient‐ and procedure‐level factors but not hospital factors (eg, procedural volume). FTR, in contrast, occurs only after an AE, and we expected it to be less influenced by case mix and more affected by the quality of the care team. As expected, the odds of FTR was associated with a smaller number of patient factors (that appeared to independently affect the risk of progressing from a relatively minor AE to a catastrophic one). Rankings of hospitals using case‐mix–adjusted AEs and rankings based on FTR were not correlated. Finally, rankings of hospitals by RSR for FTR were correlated with PCCL volume (a potential marker of hospital quality), whereas those for AEs were not. These findings suggest that FTR provides information about hospital performance not captured by case‐mix–adjusted AE and should be included in future research and quality‐improvement projects.
From a conceptual standpoint, several properties of FTR make it well suited as a measurement of quality for PCCL programs. The case mix in PCCL programs is complex, not only because of a broad range of patient characteristics (cardiac diagnoses, noncardiac conditions, and hemodynamic instability) but also because of a broad range of procedure‐associated risk.3, 4, 5, 7 The risk of AEs is strongly associated with these factors. FTR is associated with a much narrower range of these factors but is associated with procedural volume, a potential marker of programmatic quality. As a factor relatively independent of case mix, FTR may provide better discrimination of programmatic quality. This idea is supported by the fact that rankings based on risk‐adjusted pooled AEs do not correlate with those based on FTR, but rankings based on catastrophic AEs do. A possible explanation is that a greater number of AEs included in the composite outcomes are relatively minor. These relatively minor AEs are associated with the specific patient and procedure but do not appear to be associated with hospital characteristics. Even after case‐mix adjustment, these AEs (and thus composite AEs) may continue to be more reflective of case mix than FTR or catastrophic adverse outcomes.
To date, there has been greater experience in using case‐mix–adjusted AEs as a metric for adult and pediatric or congenital cardiac surgery. However, several concerns have been raised about the use of risk‐adjustment models to evaluate and report the quality of congenital heart surgery programs. First, risk‐adjustment models that are derived using large data sets may have excellent test characteristics but perform less well in discriminating between individual programs.34, 35, 36 There is a possibility of a ceiling effect for the highest risk patients, who are not captured in even the best risk‐adjustment model.35, 36 Even detailed models may not be able to account for all possible risk factors (and combinations of risk factors that may interact). The proposed FTR metric excludes the highest risk patients (emergent and salvage procedures) to avoid some of these concerns. There are also concerns that risk models derived using large data sets will have unreliable results when event rates are small for individual procedures.34, 35, 36 Finally, judging hospitals that do not engage in the entire spectrum of risk based on their performance with less risky patients or procedures produces an illusory estimate of their quality, as excellent performance in a lower risk pool does not predict similar results for higher risk cases.34, 35, 36 The proposed FTR metric studies the progress from AEs to a catastrophic AE and (as demonstrated in the current study) is relatively independent of patient‐ and procedure‐defined risk profiles. Consequently, FTR may be less subject to differences in hospital case mix than pooled AE.
A normative benefit of using FTR as a quality metric is that it encourages accurate reporting of relatively minor AEs, which might otherwise be reported differentially. This is because FTR removes a disincentive for reporting these (potentially equivocal) events and, because they are in the denominator of FTR, actually incentivizes reporting them. Catastrophic AEs are less equivocal, and reporting of them is less likely to be subjective and affected by incentives imposed by sharing of outcomes and potential public reporting. More important, using FTR as an outcome also ameliorates the disincentive to take on high‐risk or complex interventions and/or medically complex patients, who are associated with AE risk out of proportion to that predicted by risk‐adjustment models. Use of an FTR‐based ranking system places the emphasis of the metric on whether patients can be successfully shepherded through the PCCL without an AE progressing to a more catastrophic one, which is in greater alignment with the values of the field as a whole.
Our modified FTR metric also addresses the issue of attributability of postprocedure mortality, which has been questioned in the literature.28 PCCL patients differ from adult general surgical patients both in terms of the severity of their medical disease and the fact that these patients frequently undergo cardiothoracic surgery. Evaluating attributability is possible with review of the chart, but prospective adjudication of these events is challenging. In the IMPACT Registry, most of the component events in the current pooled AE composite outcome are recorded without regard to attributability. To improve the attributability of FTR to events in the catheterization laboratory, the proposed FTR metric counts deaths only if they occur within 2 days of catheterization and incorporates severe AEs that occur in combination with a more minor event in the catheterization laboratory or that were explicitly judged to be due to an event in the catheterization laboratory. We acknowledge that this strategy does not solve the issue of attributability, but it is a potential improvement over the current approach.
A factor that limited our ability to test the relative merits of composite AEs, FTR, and catastrophic AEs as quality metrics was the lack of validated hospital‐ or program‐level markers of quality. In the absence of any gold standard measures of hospital quality, we sought to use hospital annual PCCL volume as a potential marker of quality because procedural volume has been associated with superior outcomes across a broad swath of medical fields.29 Studies evaluating the volume–outcome relationship in PCCL have had equivocal results. A significant association between increasing procedural volume and reduced risk of catastrophic adverse outcomes has been demonstrated in several studies,25, 26 but when the composite outcome used included a broader range of less severe outcomes, the association was not seen.27 To clarify this, in the current study, we separately analyzed the associations between volume and risks of (1) FTR, (2) all AEs, and (3) catastrophic AEs. A significant association was demonstrated between procedural volume and the risk of FTR or catastrophic outcomes but not all AEs. Furthermore, there was a significant correlation of the ranks of hospitals by procedural volume with standardized RSR for FTR (and no such association was seen with RSR for all AEs). This finding by itself does not demonstrate that FTR is a superior outcome measure to all case‐mix–adjusted AEs. However, the association suggests that FTR has a stronger association with programmatic quality because it appears to be influenced by center‐level factors rather than patient‐ and procedure‐level factors. As the field progresses, in addition to comparing programs, future research should identify programmatic practices associated with improved outcomes. Future research evaluating other aspects of experience (eg, operator experience) would be useful but are not feasible at this time. At this point, we would recommend that both FTR and composite AEs be used in these studies.
The finding that hospitals with large proportions of adult patients were associated with reduced FTR was not anticipated. These hospitals are likely general hospitals with catheterization laboratory volume that includes coronary and structural heart cases (which are not recorded in IMPACT) on top of their recorded PCCL (underestimating their “true” volume). The observed association may be a correction for this underestimation of volume or some other factor (eg structures or processes present in institutions that perform other structural interventions such as transcatheter aortic valve replacement). Further research is necessary to clarify the factors from this association arises.
Further research is also necessary to evaluate the relative merits of FTR and traditional metrics of quality. Several other large data sets contain detailed information about PCCL procedures. Reevaluating this model in those databases may be a useful step in demonstrating its validity.
This study has several other limitations. It was an observational retrospective study, and causal inference is limited in this design. Identified associations do not imply causation. Moreover, there is always the risk of unmeasured confounding. Another concern regarding the implementation of FTR as a quality metric is that FTR is a relatively rare event, making quarterly reporting challenging. However, we would contend that for measuring the quality of a hospital program, it is paramount to find a metric with maximum utility and then to measure it over an adequate period to measure performance precisely. In addition, using catastrophic AEs (instead of FTR) does not increase statistical power (which depends on the number of events, which is similar between catastrophic AEs and FTR, not the rate of occurrence). As for quality‐improvement efforts, low event rates would make it a less agile measure. At the level of an individual catheterization laboratory, consideration of both FTR and a carefully selected pool of significant AEs may be the best means of maintaining quality. Finally, the study was performed using the IMPACT Registry, and the definitions of exposures, outcomes, and covariates are defined by those available in the database. As new versions of the registry are developed, updating the FTR metric to reflect available data (especially about outcomes) will be important.
Despite these limitations, we conclude that, as an outcome metric, FTR provides additional information that is not contained in case‐mix–adjusted AEs. FTR is complementary to case‐mix–adjusted AEs, offering a distinct perspective on identifying high‐ and low‐performing centers. We conclude that FTR should be included along with pooled AEs in future research and quality‐improvement efforts.
Sources of Funding
O'Byrne receives research support from the National Institutes of Health, National Heart, Lung, and Blood Institute (HL130420‐01), which also provided direct funding for this project. The funding agencies had no role in drafting the article or influencing its content. The proposed project and article were reviewed by the IMPACT Research and Publications Committee. The article does not represent the opinion of the funding agency, the IMPACT Research and Publications Committee, the American College of Cardiology Foundation, or the National Cardiovascular Data Registry. It is solely reflective of the views of the study authors.
Disclosures
O'Byrne has received honoraria from Gore Medical (Newark, Delaware). The remaining authors have no relevant disclosures to report.
Supporting information
(J Am Heart Assoc. 2019;8:e013151 DOI: 10.1161/JAHA.119.013151.)
References
- 1. Moore JW, Vincent RN, Beekman RH, Benson L, Bergersen L, Holzer R, Jayaram N, Jenkins K, Li Y, Ringel R, Rome J, Martin GR; NCDR IMPACT Steering Committee . Procedural results and safety of common interventional procedures in congenital heart disease: initial report from the National Cardiovascular Data Registry. J Am Coll Cardiol. 2014;64:2439–2451. [DOI] [PubMed] [Google Scholar]
- 2. Vincent RN, Moore J, Beekman RH, Benson L, Bergersen L, Holzer R, Jayaram N, Jenkins K, Ringel R, Rome J, Martin GR. Procedural characteristics and adverse events in diagnostic and interventional catheterisations in paediatric and adult CHD: initial report from the IMPACT Registry. Cardiol Young. 2016;26:70–78. [DOI] [PubMed] [Google Scholar]
- 3. Bergersen L, Gauvreau K, Jenkins KJ, Lock JE. Adverse event rates in congenital cardiac catheterization: a new understanding of risks. Cong Heart Dis. 2008;3:90–105. [DOI] [PubMed] [Google Scholar]
- 4. Bergersen L, Gauvreau K, Marshall A, Kreutzer J, Beekman R, Hirsch R, Foerster S, Balzer D, Vincent J, Hellenbrand W, Holzer R, Cheatham J, Moore J, Lock J, Jenkins K. Procedure‐type risk categories for pediatric and congenital cardiac catheterization. Circ Cardiovasc Interv. 2011;4:188–194. [DOI] [PubMed] [Google Scholar]
- 5. Bergersen L, Marshall A, Gauvreau K, Beekman R, Hirsch R, Foerster S, Balzer D, Vincent J, Hellenbrand W, Holzer R, Cheatham J, Moore J, Lock J, Jenkins K. Adverse event rates in congenital cardiac catheterization ‐ a multi‐center experience. Catheter Cardiovasc Interv. 2010;75:389–400. [DOI] [PubMed] [Google Scholar]
- 6. Bergersen L, Gauvreau K, Lock JE, Jenkins KJ. A risk adjusted method for comparing adverse outcomes among practitioners in pediatric and congenital cardiac catheterization. Cong Heart Dis. 2008;3:230–240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Bergersen L, Gauvreau K, Foerster SR, Marshall AC, McElhinney DB, Beekman RH, Hirsch R, Kreutzer J, Balzer D, Vincent J, Hellenbrand WE, Holzer R, Cheatham JP, Moore JW, Burch G, Armsby L, Lock JE, Jenkins KJ. Catheterization for Congenital Heart Disease Adjustment for Risk Method (CHARM). JACC Cardiovasc Interv. 2011;4:1037–1046. [DOI] [PubMed] [Google Scholar]
- 8. Holzer RJ, Gauvreau K, Kreutzer J, Moore JW, McElhinney DB, Bergersen LJ. Relationship between procedural adverse events associated with cardiac catheterization for congenital heart disease and operator factors. Cathet Cardiovasc Intervent. 2013;82:463–473. [DOI] [PubMed] [Google Scholar]
- 9. Silber JH, Williams SV, Krakauer H, Schwartz JS. Hospital and patient characteristics associated with death after surgery. A study of adverse occurrence and failure to rescue. Med Care. 1992;30:615–629. [DOI] [PubMed] [Google Scholar]
- 10. Silber JH, Rosenbaum PR, Ross RN. Comparing the contributions of groups of predictors: which outcomes vary with hospital rather than patient characteristics. J Am Stat Assoc. 1995;90:7–18. [Google Scholar]
- 11. Silber JH, Rosenbaum PR, Schwartz JS, Ross RN, Williams SV. Evaluation of the complication rate as a measure of quality of care in coronary artery bypass graft surgery. JAMA. 1995;274:317–323. [PubMed] [Google Scholar]
- 12. Silber JH, Romano PS, Rosen AK, Wang Y. Failure‐to‐rescue: comparing definitions to measure quality of care. Med Care. 2007;45:918–925. [DOI] [PubMed] [Google Scholar]
- 13. Waits SA, Sheetz KH, Campbell DA, Ghaferi AA, Englesbe MJ, Eliason JL, Henke PK. Failure to rescue and mortality following repair of abdominal aortic aneurysm. J Vasc Surg. 2014;59:909–914.e1. [DOI] [PubMed] [Google Scholar]
- 14. Ghaferi AA, Osborne NH, Birkmeyer JD, Dimick JB. Hospital characteristics associated with failure to rescue from complications after pancreatectomy. J Am Coll Surg. 2010;211:325–330. [DOI] [PubMed] [Google Scholar]
- 15. Ghaferi AA, Birkmeyer JD, Dimick JB. Complications, failure to rescue, and mortality with major inpatient surgery in medicare patients. Ann Surg. 2009;250:1029–1034. [DOI] [PubMed] [Google Scholar]
- 16. Ghaferi AA, Dimick JB. Variation in mortality after high‐risk cancer surgery: failure to rescue. Surg Oncol Clin N Am. 2012;21:389–395, vii. [DOI] [PubMed] [Google Scholar]
- 17. Shah R, Attwood K, Arya S, Hall DE, Johanning JM, Gabriel E, Visioni A, Nurkin S, Kukar M, Hochwald S, Massarweh NN. Association of frailty with failure to rescue after low‐risk and high‐risk inpatient surgery. JAMA Surg. 2018;153:e180214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Kuo LE, Kaufman E, Hoffman RL, Pascual JL, Martin ND, Kelz RR, Holena DN. Failure‐to‐rescue after injury is associated with preventability: the results of mortality panel review of failure‐to‐rescue cases in trauma. Surgery. 2017;161:782–790. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Chung JJ, Earl‐Royal EC, Delgado MK, Pascual JL, Reilly PM, Wiebe DJ, Holena DN. Where we fail: location and timing of failure to rescue in trauma. Am Surg. 2017;83:250–256. [PMC free article] [PubMed] [Google Scholar]
- 20. Earl‐Royal E, Kaufman EJ, Hsu JY, Wiebe DJ, Reilly PM, Holena DN. Age and preexisting conditions as risk factors for severe adverse events and failure to rescue after injury. J Surg Res. 2016;205:368–377. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Holena DN, Kaufman EJ, Delgado MK, Wiebe DJ, Carr BG, Christie JD, Reilly PM. A metric of our own. J Trauma Acute Care Surg. 2017;83:698–704. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Silber JH, Arriaga AF, Niknam BA, Hill AS, Ross RN, Romano PS. Failure‐to‐rescue after acute myocardial infarction. Med Care. 2018;56:416–423. [DOI] [PubMed] [Google Scholar]
- 23. Pasquali SK, He X, Jacobs JP, Jacobs ML, O'Brien SM, Gaynor JW. Evaluation of failure to rescue as a quality metric in pediatric heart surgery: an analysis of the STS Congenital Heart Surgery Database. Ann Thorac Surg. 2012;94:573–579; discussion 579–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Messenger JC, Ho KKL, Young CH, Slattery LE, Draoui JC, Curtis JP, Dehmer GJ, Grover FL, Mirro MJ, Reynolds MR, Rokos IC, Spertus JA, Wang TY, Winston SA, Rumsfeld JS, Masoudi FA; NCDR Science and Quality Oversight Committee Data Quality Workgroup . The National Cardiovascular Data Registry (NCDR) Data Quality Brief: the NCDR Data Quality Program in 2012. J Am Coll Cardiol. 2012;60:1484–1488. [DOI] [PubMed] [Google Scholar]
- 25. O'Byrne ML, Glatz AC, Shinohara RT, Jayaram N, Gillespie MJ, Dori Y, Rome JJ, Kawut S. Effect of center catheterization volume on risk of catastrophic adverse event after cardiac catheterization in children. Am Heart J. 2015;169:823–832.e5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. O'Byrne ML, Glatz AC, Hanna BD, Shinohara RT, Gillespie MJ, Dori Y, Rome JJ, Kawut SM. Predictors of catastrophic adverse outcomes in children with pulmonary hypertension undergoing cardiac catheterization: a multi‐institutional analysis from the pediatric health information systems database. J Am Coll Cardiol. 2015;66:1261–1269. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Jayaram N, Spertus JA, O'Byrne ML, Chan PS, Kennedy KF, Bergersen L, Glatz AC. Relationship between hospital procedure volume and complications following congenital cardiac catheterization: a report from the IMproving Pediatric and Adult Congenital Treatment (IMPACT) registry. Am Heart J. 2017;183:118–128. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Backes CH, Bergersen L, Rome JJ, Batlivala SP, Glatz AC, Ovunc B, David S, Rivera BK, Haque U, Kollins K, Yin H, Holzer RJ. Quality metrics in cardiac catheterization for congenital heart disease: utility of 30‐day mortality. Catheter Cardiovasc Interv. 2015;85:104–110. [DOI] [PubMed] [Google Scholar]
- 29. Halm EA, Lee C, Chassin MR. Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Int Med. 2002;137:511–520. [DOI] [PubMed] [Google Scholar]
- 30. Jayaram N, Spertus JA, Kennedy KF, Vincent R, Martin GR, Curtis JP, Nykanen D, Moore PM, Bergersen L. Modeling major adverse outcomes of pediatric and adult patients with congenital heart disease undergoing cardiac catheterization: observations from the NCDR IMPACT Registry (National Cardiovascular Data Registry Improving Pediatric and Adult Congenital Treatment). Circulation. 2017;136:2009–2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. O'Byrne ML, Kennedy KF, Rome JJ, Glatz AC. Variation in practice patterns in device closure of atrial septal defects and patent ductus arteriosus: an analysis of data from the IMproving Pediatric and Adult Congenital Treatment (IMPACT) registry. Am Heart J. 2018;196:119–130. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Glatz AC, Kennedy KF, Rome JJ, O'Byrne ML. Variations in Practice Patterns and Consistency With Published Guidelines for Balloon Aortic and Pulmonary Valvuloplasty. JACC Cardiovasc Interv. 2018;11:529–538. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. O'Byrne ML, Kennedy KF, Kanter JP, Berger JT, Glatz AC. Risk factors for major early adverse events related to cardiac catheterization in children and young adults with pulmonary hypertension: an analysis of data from the IMPACT (Improving Adult and Congenital Treatment) Registry. J Am Heart Assoc. 2018;7:e008142 DOI: 10.1161/JAHA.117.008142. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Shahian DM, Jacobs JP, Badhwar V, D'Agostino RS, Bavaria JE, Prager RL. Risk aversion and public reporting. part 2: mitigation strategies. Ann Thorac Surg. 2017;104:2102–2110. [DOI] [PubMed] [Google Scholar]
- 35. Spray TL, Gaynor JW. A word of caution in public reporting. Semin Thorac Cardiovasc Surg: Pediatr Card Surg Annu. 2017;20:49–55. [DOI] [PubMed] [Google Scholar]
- 36. Gaynor JW, Pasquali SK, Ohye RG, Spray TL. Potential benefits and consequences of public reporting of pediatric cardiac surgery outcomes. J Thorac Cardiovasc Surg. 2017;153:904–907. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.