The Impact of Extreme-Risk Cases on Hospitals’ Risk-Adjusted Percutaneous Coronary Intervention Mortality Ratings

Abstract

OBJECTIVES

The goal of this study was to examine the calibration of a validated risk-adjustment model in very high-risk percutaneous coronary intervention (PCI) cases and assess whether sites’ case mix affects their performance ratings.

BACKGROUND

There are concerns that treating PCI patients with particularly high-risk features such as cardiogenic shock or prior cardiac arrest may adversely impact hospital performance ratings. However, there is little investigation on the validity of these concerns.

METHODS

We examined 624,286 PCI procedures from 1,168 sites that participated in the CathPCI Registry in 2010. Procedural risk was estimated using the recently published Version 4 National Cardiovascular Data Registry (NCDR) PCI risk-adjusted mortality (RAM) model. We calculated observed/expected mortality using several risk classification methods, and simulated hospital performance after combining their highest risk cases over 2 years into a single year.

RESULTS

In 2010, crude in-hospital PCI mortality was 1.4%. The NCDR model was generally well calibrated among high risk, however there was slight overprediction of risk in extreme cases. Hospitals treating the highest overall expected risk PCI patients or those treating the top 20% of high-risk cases had lower (better) RAM ratings than centers treating lower-risk cases (1.25% vs. 1.51%). The observed/expected ratio for top-risk quintile versus low-risk quintile was 0.91 (0.87 to 0.96) versus 1.10 (1.03 to 1.17). Combining all the high-risk patients over a 2-year period into a single year also did not negatively impact the site’s RAM ratings.

CONCLUSIONS

Evaluation of a contemporary sample of PCI cases across the United States showed no evidence that treating high-risk PCI cases adversely affects hospital RAM rates.

Hospital quality is now judged by several metrics. A common measure for hospital outcomes is risk-adjusted mortality (RAM) (1,2), which is calculated for many different clinical conditions, including percutaneous coronary intervention (PCI). Historically, these risk-adjustment models have their basis in the belief that mortality, if appropriately adjusted for case mix, is a measure of overall hospital quality (3). Hospitals and providers have voiced concerns that risk-adjustment models employed to account for case mix may not adequately account for particularly high-risk clinical features (4,5), and that clinicians and hospitals treating a greater number of high-risk patients may have a worse rating (6). At worst, these concerns might lead clinicians to avoid very high-risk, but appropriate-to-treat, patients in order to protect their RAM ratings (7,8).

PCI is a commonly performed procedure, occurring at approximately 1,700 medical centers in the United States (9). These centers have a wide variation in hospital volume and case mix (10), hence RAM ratings are a commonly used, though controversial, quality measure. Cardiac arrest and cardiogenic shock patients represent the highest-risk patients potentially treated by PCI. These 2 groups of patients have the highest in-hospital mortality, but may also have the highest potential benefit from urgent percutaneous revascularization (11–15). PCI practice patterns in the United States show significant variability for patients with cardiac arrest and cardiogenic shock, and recent data indicate that public reporting of performance measures may partially drive the differences seen in these practice patterns (16). A possible explanation for these data is that PCI practitioners may change their behavior (become more risk avoidant) to avoid a negative impact on their RAM ratings. There has been significant documentation of these concerns (17) and even observational evidence of shifts in PCI and surgical practice patterns (8,18).

Our study aimed to evaluate the accuracy of a validated and widely used risk adjustment model from the National Cardiovascular Data Registry (NCDR) CathPCI registry, the current NCDR PCI RAM model, to estimate mortality in moderate- and high-risk subsets. We then aimed to assess whether sites treating more high-risk cases have worse observed versus expected mortality ratios and worse RAM ratlings than sites treating lower-risk patients.

METHODS

The CathPCI Registry is a of the American College of Cardiology Foundation and the Society for Cardiovascular Angiography and Interventions, and remains the largest ongoing registry of PCI in the United States. Descriptions of the registry have been previously published (9). The registry collects data on patient and hospital characteristics, clinical presentation, procedural characteristics, and in-hospital outcomes for PCI procedures from >1,200 sites across the United States. Data are entered into NCDR-certified software at participating institutions, and exported in a standard format to the American College of Cardiology. There is a comprehensive data quality program, including both data quality report specifications for data capture and transmission, as well as an auditing program (19).

STUDY POPULATION

All PCI procedures in the NCDR CathPCI registry in calendar year 2010 were initially included. We excluded those procedures that were repeat procedures in a patient, that were transferred to another facility after the procedure, those with unknown vital status, and those hospitals whose annual volume was <50 PCIs. This resulted in a final study population of 624,286 PCI procedures at 1,168 hospitals (Figure 1).

FIGURE 1. Study Sample Selection Diagram.

The flow diagram shows the derivation of the study population. NCDR = National Cardiovascular Data Registry; PCI = percutaneous coronary intervention.

OUTCOMES AND CURRENT RISK MODEL

The primary outcome evaluated in this study was in-hospital mortality. The current, Version 4 NCDR PCI RAM model has been recently described in detail (20). Briefly, the current model was based upon the findings in over 1 million PCI patients seen between mid-2009 and mid-2011, and includes 18 clinical variables. It was developed in 60% of the total population, and validated in the remaining 40% of the study population. The model showed excellent discriminative ability, with a c-index of 0.93. It also showed good calibration with a Hosmer-Lemeshow goodness-of-fit statistics p value of <0.05.

STATISTICAL METHODS

As previously described, the predicted in-hospital mortality risk was calculated for all patient visits in the sample, and the distribution of patient risk was plotted. High-risk patients were prospectively defined as those with an expected mortality rate of ≥10%. Model calibration was evaluated by plotting the expected versus the observed mortality in deciles of expected mortality. Further evaluation of model calibration was performed by plotting the expected versus observed mortality for the subset of cardiogenic shock and/or salvage PCI patients, defined as cardiogenic shock at the start of the procedure, requiring manual compression for cardiac arrest within 10 min of the procedure start, or the requirement of extracorporeal hemodynamic support at the beginning of the procedure.

The hospital-level predicted RAM was then calculated by combining the patient predicted mortality risk for all patients treated at a hospital site. The hospital sites were grouped into risk quintiles by overall hospital expected mortality; patient characteristics and hospital characteristics were reported for each quintile. The ratio of observed versus expected mortality was calculated with 95% confidence intervals (CIs; based on binomial distribution) for each quintile of risk. Sensitivity analyses were also performed by repeating this approach with hospitals ranked in quintiles by the percent of cardiogenic shock/cardiac arrest patients treated, and by the percentage of high-risk (≥ 10% predicted mortality) patients treated.

As a third method of analysis, sites’ highest-risk cases from a 2-year period (2009 2 through 2011 2) were combined into a single year’s volume to simulate a “concentrated risk year,” and the hospital sites were used as their own control. This was done by rank ordering the patients according to their predicted risk of mortality, selecting the hospitals’ highest-risk cases, and assuming that these patients were seen in a single year, keeping annual case volume constant. Observed versus expected mortality (O/E) ratios were again calculated with 95% CIs for each quintile based on expected risk. The O/E ratio in the concentrated risk year scenario was then plotted against the average O/E ratio during that time period, with 95% CIs. Centers whose 95% CI for observed mortality was significantly better or worse than for the predicted mortality were flagged as outliers. The frequency of outlier status was then compared between the concentrated risk year and the overall 2-year period. All statistical analyses were performed by the Duke Clinical Research Institute using SAS software (version 9.2, SAS Institute, Cary, North Carolina). The study was reviewed by the Duke Institutional Review Board and was determined to meet the definition of research not requiring informed consent, given that patient information is collected anonymously without unique patient identifiers and only aggregate data are presented and published.

RESULTS

From January 2010 to December 2010, 647,210 PCI procedures during 630,377 admissions in 1,208 sites were registered in the CathPCI registry. After application of the aforementioned exclusion criteria, 624,286 PCI procedures in 1,168 sites were included in the analysis (Figure 1). The CathPCI mortality model (V4) demonstrated good performance over the spectrum of risk (from 0.01% to 100%) (Figure 2A). The model performance seemed equally good when the same O/E ratios were calculated in the extremely high-risk subgroups of patients with cardiogenic shock and/or salvage PCI and in patients at extreme mortality risk (>10%) (Figure 2B).

FIGURE 2. Calibration Plots for Mortality Risk.

Calibration plots for the full spectrum of mortality risk (A) and in high-risk subgroups of cardiac arrest and salvage percutaneous coronary intervention (PCI) patients (B) are shown. The observed versus expected mortality plots show good calibration of the CathPCI V4 model for the full spectrum of risk in PCI patients. Carshock = cardiogenic shock.

The baseline clinical characteristics of the patient population stratified by the hospital risk quintiles are shown in Table 1. Patients at higher-risk hospitals were of similar age, race, and sex, and had similar frequency of insulin-dependent diabetes, though were less likely to have peripheral arterial disease, cerebrovascular disease, prior congestive heart failure compared with those in lower-risk hospitals. However, patients at the highest-risk hospitals were more likely to have prior myocardial infarction and to present with ST-segment elevation myocardial infarction, New York Heart Association functional class IV heart failure symptoms, shock, or cardiac arrest. Also, the proportion of PCI procedures at high-risk hospitals was less likely to be elective, more likely emergent, and more often involved the proximal left anterior descending coronary artery when compared with lower-risk hospitals. The hospital-level characteristics, grouped by hospital risk quintile, are shown in Table 2. High-risk hospitals, compared with low-risk hospitals, were larger (by bed number), had a lower annual PCI volume, and were more likely to be rural or academic centers. There was substantial variation present in the geographic distribution of hospitals by risk quintile. A greater proportion of hospitals in the West are in the highest-risk quintile group, whereas a greater proportion of Southern hospitals are in the lowest-risk quintile group, when compared with the Northeast and Midwest regions (Table 3).

TABLE 1.

Baseline Characteristics by Predicted Mortality Quintile

	Hospital Risk Groups (Quintiles)
	Q1 Highest	Q2	Q3	Q4	Q5 Lowest
Age, yrs	64.5 ± 12.4	64.9 ± 12.2	64.8 ± 12.1	64.5 ± 12.0	64.7 ± 11.9
Female	31.5	31.8	32.3	33.2	34.1
Caucasian	87.8	89.6	88.9	87.8	88.7
Prior MI	30.4	31.5	29.8	29.8	28.3
Prior CHF	12.3	12.3	11.3	11.5	11.3
IDDM	12.8	12.6	12.9	13.6	12.9
ESRD (HD)	2.9	2.3	2.3	2.3	1.8
PAD	11.9	12.7	12.4	12.7	12.8
GFR (CG)	73.9 ± 28.8	75.0 ± 29.9	75.6 ± 29.6	75.4 ± 32.0	76.7 ± 31.0
STEMI	24.0	17.8	16.2	13.9	10.6
NYHA IV	6.4	4.8	4.0	3.2	2.0
Shock, 24 h	3.6	2.3	1.8	1.3	0.8
Cardiac arrest	3.2	2.2	1.8	1.4	0.9
Lesion
Left Main	1.7	1.8	1.7	1.8	1.6
Prox. LAD	16.3	15.2	15.0	14.5	14.3
PCI Status
Elective	32.6	39.7	42.6	47.9	60.5
Urgent	41.1	40.8	39.9	37.2	28.3
Emergent	25.5	19.1	17.3	14.8	11.1

Values are % or mean ± SD. Baseline characteristics, comorbidities, admission data, and laboratory studies, stratified by Hospital risk Quintile.

CHF = congestive heart failure; ESRD (HD) = end-stage renal disease (hemodialysis); GFR (CG) = glomerular filtration rate (Cockcroft-Gault); IDDM = insulin-dependent diabetes mellitus; LVEF = left-ventricular ejection fraction; MI = myocardial infarction; NYHA = New York Heart Association functional class; PAD = peripheral artery disease; PCI = percutaneous coronary intervention; Prox. LAD = proximal left anterior descending coronary artery; STEMI = ST-segment elevation myocardial infarction.

TABLE 2.

Hospital Characteristics by Predicted Mortality Quintile

	Hospital Risk Groups (Quintiles)
	Q1 Highest	Q2	Q3	Q4	Q5 Lowest
Bed number	470 ± 241	472 ± 234	494 ± 393	472 ± 240	370 ± 225
PCI volume	591 ± 404	764 ± 406	830 ±515	1,067 ± 775	1,015 ± 682
Rural	11.7	10.5	14.5	10.3	9.2
Urban	61.8	55.5	55.5	59.6	61.1
Academic	52.0	52.0	49.8	54.5	41.4

Values are % or mean ± SD.

PCI = percutaneous intervention.

TABLE 3.

Hospitals Predicted Mortality Quintile by Regions

	Hospital Risk Groups (Quintiles)
	Q1 Highest	Q2	Q3	Q4	Q5 Lowest
West (n = 96,764)	23.8	27.5	27.2	14.0	7.4
Northeast (n = 89,723)	12.0	28.6	17.7	22.7	18.5
Midwest (n = 183,540)	12.2	22.9	22.4	24.4	18.1
South (n = 255,333)	8.9	16.0	19.9	28.3	26.8

Values are %.

On comparison, the O/E ratios for each risk quintile of hospitals were similar and approximately 1.0, except for the highest-risk quintile of hospitals, for which the O/E ratio was 0.91 (95% CI: 0.87 to 0.96), indicating these hospitals performed better than expected (Table 4). In addition, when comparing hospital-level RAM, the highest-risk hospitals had slightly lower RAM than the lowest-risk hospitals. A sensitivity analysis, in which hospital sites were regrouped by frequency of patients with an expected mortality risk of ≥10%, found similar results, with the hospitals’ RAM rates being slightly lower in the highest-risk hospitals.

TABLE 4.

Observed Versus Expected Mortality and RAM by Predicted Mortality Quintile

	Sites, N	Observed Mortality	Expected Mortality	O/E Ratio (95% CI)	RAM (95% CI)
Overall	1,168	1.37%	1.37%	1.00 (0.98–1.02)	1.37% (1.34–1.40)
Top 20%	233	2.21%	2.43%	0.91 (0.87–0.96)	1.25% (1.20–1.31)
Top 20%–40%	234	1.63%	1.69%	0.96 (0.92–1.00)	1.32% (1.27–1.38)
Middle 20%	234	1.39%	1.36%	1.02 (0.98–1.07)	1.41% (1.34–1.47)
Bottom 20%–40%	234	1.16%	1.08%	1.07 (1.02–1.12)	1.47% (1.40–1.54)
Bottom 20%	233	0.81%	0.74%	1.10 (1.03–1.17)	1.51% (1.42–1.60)

CI = confidence interval; O/E ratio = observed versus expected mortality ratio; RAM = risk-adjusted mortality.

As a final analysis, we compared the overall performance of the hospitals in an extreme scenario, the concentrated risk year (Table 5). For all risk quintiles, the O/E ratio was approximately 1.0 with CIs that crossed the line of unity, and the agreement of concentrated risk year O/E with average year O/E was consistent across nearly all centers (Figure 3). In addition, during the overall 2009 and 2011 period, 65 (5.4%) centers were flagged as lower-performing outliers, whereas 48 (4.0%) were classified as higher-performing outliers. During the concentrated risk year, 60 (5.0%) were flagged as lower performing, and 45 (3.7%) as higher-performing outliers, indicating no increase in the identification of outlier hospitals when creating a higher-risk annual case mix.

TABLE 5.

Concentrated Risk Year Versus Average Year O/E Ratios by Predicted Mortality Risk Quintile

	Sites, N	O/E Concentrated Risk Year (95% CI)	O/E Overall (95% CI)	O/E:O/E (95% CI)
Overall	1,204	1.02 (1.00–1.03)	1.00 (0.99–1.02)	1.01 (0.99–1.04)
Top 20%	241	0.98 (0.95–1.00)	0.96 (0.94–0.98)	1.01 (0.98–1.05)
Top 20%–40%	241	1.00 (0.97–1.03)	0.99 (0.96–1.02)	1.01 (0.97–1.06)
Middle 20%	241	1.09 (1.05–1.13)	1.08 (1.04–1.12)	1.01 (0.96–1.06)
Bottom 20%–40%	241	1.09 (1.04–1.14)	1.08 (1.02–1.13)	1.01 (0.94–1.09)
Bottom 20%	240	1.16 (1.07–1.26)	1.15 (1.05–1.24)	1.01 (0.90–1.14)

Abbreviations as in Table 4.

FIGURE 3. Observed/Expected Mortality by Hospital Risk Quintile in a “Concentrated Risk Year” Versus an Average Year.

Observed versus expected mortality (O/E) ratios when the highest-risk cases or concentrated into a single year compared with overall O/E ratio in an average year show there is still a 1:1 relationship at all levels of risk. The 95% confidence intervals are shown with dotted lines, and a regression line fit to the hospital data shows consistent agreement between concentrated risk year and average year performance.

DISCUSSION

There is a growing emphasis to publically report hospitals’ performance and outcomes from common diseases. Our study showed the current CathPCI V4 mortality model performed well with good discriminative ability and calibration across the spectrum of risk. Using this model, we also found that hospitals treating more high-risk PCI patients did not have worse outcomes after risk adjustment by this means. We tested this by comparing observed/expected mortality in hospitals grouped by 3 different measures of risk, and concentrating the highest-risk cases from a 2-year period into 1 year, and subsequently comparing observed/expected mortality. In both instances, we found no evidence that RAM was worse when higher-risk patients were included.

HIGH-RISK AND EXTREME-RISK CASES

Previous investigation raised concerns that even a well-calibrated model may lack certain variables that are known to signal the “compassionate use” of PCI and thus a very high-risk group that could affect RAM ratings. Resnic et al. (21) studied this question in the MassDAC (Massachusetts Data Analysis Center) registry, defining compassionate use as either coma on presentation for emergent PCI, ventricular assist device or extracorporeal bypass, or cardiopulmonary resuscitation at the initiation of the procedure. The combination and inclusion of these variables into the standard NCDR CathPCI mortality model did improve the c-statistic for prediction of mortality (0.87 to 0.90; p < 0.001). However, the number of cases that qualified under these compassionate use criteria was very small, and the authors included a further analysis of hospital RAM classification showing that inclusion of compassionate use criteria did not significantly change any hospital’s RAM rating.

PHYSICIAN ATTITUDES AND PRACTICES

RAM as a measure of hospital quality has been criticized in many ways, but remains a widely reported metric. Physician survey data reveal concern over the effect of high-risk cases on hospital and provider RAM estimates. Narins et al. (17) published data from a survey of physicians in New York that indicates 85% of respondents believed that the RAM model was insufficient to account for case mix and that nearly 80% of physicians felt that their decision making in the case of critically ill patients in need of angiography or PCI might be affected by the presence of RAM ratings. These sentiments are borne out in recently published data from New York and Massachusetts. Moscucci et al. (8) showed that in comparison to Michigan, where there was no system of public reporting in place, New York patients undergoing PCI were less likely to have heart failure or cardiogenic shock and had lower unadjusted mortality rates. The authors proposed one explanation for this phenomenon might have been physician reluctance to perform high-risk cases because of public reporting. McCabe et al. (18) demonstrated a similar pattern of practice in Massachusetts soon after statewide mandate of public reporting of PCI RAM rates. At previously designated outlier institutions in Massachusetts, patient risk characteristics were significantly lower after initiation of public reporting than before, again indicating a possible change to risk-averse practices because of public reporting.

Our study provides no evidence to support these attitudes and practices. In hospitals across the United States, those with the highest-risk cases showed no adverse effect on their RAM. In fact, there was a signal in our analysis that hospitals that perform the highest-risk cases, by predicted mortality, may be outperforming expectations, thus have a lower than expected RAM. Our study’s results are based on over 600,000 cases from more than 1,150 sites around the United States, and provided similar results when hospital’s case mix was risk stratified using 3 different metrics. In each case, there was no adverse effect on observed/expected mortality, and RAM was reduced in the highest-risk hospitals as compared with the lowest-risk hospitals. Finally, when artificially creating a year with a very high-risk case mix, there was consistent site performance and no increase in the number of outlier hospitals identified in this population.

CLINICAL IMPLICATIONS

With the expansion of public reporting in the United States, and the potential changes in practice patterns because of these reports, our results have important implications. Dissemination of these data could potentially reassure practitioners, emphasizing that treating the highest-risk patients will not adversely affect RAM ratings. These high-risk patients may be those that benefit the most from intervention (11,12). Our results also show significant geographic variation in hospital practice patterns with regard to PCI risk. Programs of surveillance should be established with the goal of preventing a further shift in practice towards inappropriate avoidance of high-risk patients, and providing the highest-quality care across all geographic regions (4).

STUDY LIMITATIONS

Certain factors should be considered in the interpretation of our study. First, the CathPCI V4 mortality model was constructed using data from this time period, and thus would have validity for our analysis, but may not be generalizable to all patients undergoing cardiac catheterization. Also, previous CathPCI-based mortality models may not have included variables that allowed for accurate prediction in the highest-risk cases (21). The current mortality model seems to have addressed these concerns (20). The NCDR CathPCI is a large national registry that includes community and academic centers, but may not fully represent practice in all areas of the United States. However, it is estimated that the NCDR collects data from 85% to 90% of all U.S. centers and >90% of all PCIs performed (9,19). Finally, CathPCI is a voluntary program, and thus hospitals that choose to participate may be more attuned to quality measures than other hospitals. This also presents the possibility that hospitals might “up-code” patient risk to improve quality measures such as O/E ratios. There is no evidence that this occurred in our sample population.

CONCLUSIONS

We found that the current CathPCI V4 RAM model accurately predicted mortality in high-risk patient subgroups and, if anything, overestimated the hazard in very high-risk patients. We found nothing that would suggest that altering the current decision making for treating high-risk patients would result in worse demonstrated hospital RAM ratings. These findings should reassure physicians and could help to prevent inappropriate risk-avoidant behavior in PCI practice.

To address concerns that treating percutaneous coronary intervention (PCI) patients with high-risk features may adversely impact hospital performance ratings, we examined a validated risk-adjustment model in high-risk PCI cases to assess whether sites’ case mix affects their performance ratings. Our study sample included 624,286 PCI procedures at 1,168 sites from the CathPCI Registry in 2010. Crude in-hospital PCI mortality was 1.4%. The V4 NCDR PCI risk-adjusted mortality (RAM) model was well calibrated among high-risk cases. Hospitals treating the highest overall expected risk PCI patients had lower (better) RAM ratings than centers treating lower-risk cases (1.25% vs. 1.51%). Combining high-risk patients over a 2-year period into a single year did not negatively impact sites’ RAM ratings. There was no evidence that treating high-risk PCI cases adversely affects hospital RAM rates.

ABBREVIATIONS AND ACRONYMS

CI

confidence interval

NCDR

National Cardiovascular Data Registry

O/E ratio

observed versus expected mortality ratio

PCI

percutaneous coronary intervention

RAM

risk-adjusted mortality