Comparing Two Methods of Assessing 30-Day Readmissions: What is the Impact on Hospital Profiling in the Veterans Health Administration?

Abstract

Background

The Centers for Medicare and Medicaid Services’ (CMS) all-cause readmission measure and the 3M^™ Health Information System Division Potentially Preventable Readmissions (PPR) measure are both used for public reporting. These two methods have not been directly compared in terms of how they identify high- and low-performing hospitals.

Objectives

To examine how consistently the CMS and PPR methods identify performance outliers, and explore how the PPR preventability component impacts hospital readmission rates, public reporting on CMS’ Hospital Compare website, and pay-for-performance under CMS’ Hospital Readmission Reduction Program (HRRP) for three conditions (acute myocardial infarction, heart failure, and pneumonia).

Methods

We applied the CMS all-cause model and the PPR software to VA administrative data to calculate 30-day observed FY08-10 VA hospital readmission rates and hospital profiles. We then tested the effect of preventability on hospital readmission rates and outlier identification for reporting and pay-for-performance by replacing the dependent variable in the CMS all-cause model (Yes/No readmission) with the dichotomous PPR outcome (Yes/No preventable readmission).

Results

The CMS and PPR methods had moderate correlations in readmission rates for each condition. After controlling for all methodological differences but preventability, correlations increased to >90%. The assessment of preventability yielded different outlier results for public reporting in 7% of hospitals; for 30% of hospitals there would be an impact on HRRP reimbursement rates.

Conclusions

Despite uncertainty over which readmission measure is superior in evaluating hospital performance, we confirmed there are differences in CMS- and PPR-generated hospital profiles for reporting and pay-for-performance, due to methodological differences and the PPR’s preventability component.

Introduction

Thirty-day hospital readmission is a nationally recognized quality metric. Rates of 30-day readmission are publicly reported for most private sector and Veterans Health Administration (VA) hospitals, and recent health reform legislation endorsed the use of readmission rates for hospital profiling and pay-for-performance. (1–4) Various approaches exist for examining readmissions, ranging from those that focus on specific index conditions but include all readmissions regardless of circumstances of the readmission to methods that focus on all types of index cases but include only those readmissions that are judged related to the index admission and potentially preventable. (5, 6) There is little insight into the consequences of using different measures that cover the same quality subject, and thus little consensus on which method is best for assessing hospital quality. (7)

Two well-known methods to identify 30-day readmissions are the Centers for Medicare and Medicaid Services’ (CMS) all-cause readmission measure for acute myocardial infarction (AMI), heart failure (HF), and pneumonia (PNA), (5, 8–11) and the 3M^™ Health Information System Division Potentially Preventable Readmissions (PPR) measure. (12) The CMS method is used for public reporting of hospital readmissions for Medicare fee-for-service (FFS) patients on the Hospital Compare website and to adjust Medicare payments to low-performing hospitals (i.e., pay-for-performance). (13, 14) CMS Hospital Compare also reports the 30-day readmission rate in VA hospitals. (1) The PPRs are used by several insurance companies and state Medicaid programs for reporting purposes; some states also use PPR rates to adjust Medicaid payments to low-performing hospitals. (15–18)

A recent study by the Commonwealth Fund compared the CMS and PPR 30-day readmission rates among hospitals in Massachusetts (MA) and found poor correlation in hospital rankings. (19) The poor correlation between the CMS and PPR rates was largely attributed to differences in how 30-day readmission rates are calculated. These include differences in the patient population considered eligible for a potential readmission, the risk-adjustment methodology, and the definition of which hospitalizations are considered as readmissions. The principal difference, however, is that in contrast to CMS all-cause readmissions for AMI/HF/PNA, PPRs are clinically-related to the index hospitalization and considered potentially preventable. To date, no studies have examined the extent to which the preventability component explains differences between CMS and PPRs in readmission rates and hospital rankings.

Expanding upon the Commonwealth Fund research, this study compares the CMS and PPR 30-day readmission methods using a national sample of VA hospitals and examines hospital profiles generated by the two methods. Our specific objectives are to: 1) calculate hospital 30-day readmission rates using the CMS and PPR methods, and 2) isolate the role that the PPR preventability component plays in hospital reporting and pay-for-performance. This study will provide hospitals and policy makers with a better understanding of the reasons for differences in the 30-day readmission rates and hospital profiles generated by two commonly used methods.

Methods

Data Source

We used FY06-FY10 VA inpatient and outpatient administrative data from the VA’s National Patient Care Database (20) for two specific purposes: 1) to identify index admissions and readmissions between FY08-FY10, and 2) to identify risk factors used for risk adjustment in the CMS approach (we added FY06 and FY07 inpatient and outpatient data to our sample for this purpose).

The VA Boston Healthcare System Institutional Review Board (IRB) approved this study.

CMS 30-day Risk-Adjusted All-Cause Readmission Methods for AMI/HF/PNA

Identification of Index Admissions

A total of 1,445,221 acute admissions during FY08-10 were used to identify AMI/HF/PNA index admission cohorts. We followed the CMS methodology in each stage of calculating the AMI/HF/PNA readmission rates. (8–10) For each cohort, if the same patient had contiguous AMI/HF/PNA hospitalizations (i.e., on the same or next day to the same hospital), we combined these into a single episode of care; if there was a transfer from one hospital to another, the receiving hospital admission was considered the index admission (though for risk adjustment, diagnostic codes from the transferring hospital were used). We then excluded hospitalizations with a principal diagnosis other than AMI/HF/PNA. We also excluded the following: in-hospital deaths, patients who left against medical advice, and AMI patients discharged on the day of admission. For patients with multiple cohort admissions (i.e., AMI/HF/PNA) within 30 days, we considered only the first one as an index admission because the latter cases were potential readmissions.

Identification of Readmissions

The CMS methodology identifies the first all-cause readmission within 30 days of the index discharge. However, “planned” readmissions for a percutaneous coronary intervention (PCI) or coronary artery bypass graft (CABG) procedure that following an index AMI admission are excluded, except for readmissions with principal diagnoses that are clearly not consistent with an elective readmission (e.g., HF, AMI, unstable angina, arrhythmia, and cardiac arrest). (For the HF and PN, there are no exclusions for planned readmissions.)

Risk Adjustment

We used both the principal and secondary diagnoses from the inpatient stays and provider face-to-face outpatient encounters during the 12 month pre-index period, as well as secondary diagnoses from the index admission to determine risk-adjustment covariates. The CMS methodology first groups 15,000+ ICD-9-CM diagnosis codes into one of 804 mutually exclusive groupings (DxGroups) and then aggregates these into 189 condition categories (CCs) which are used as variables for risk adjustment. Specific to AMI, the risk factors “history of PCI” and “history of CABG” were identified based on procedure codes in pre-index admissions.

Calculation of Risk-Adjusted Readmission Rates

Following the CMS approach, for each cohort, 1) we used hierarchical generalized linear models (HGLMs) to estimate coefficients for the risk factors included in the CMS model and also hospital random effects; 2) from the model, we calculated the predicted probability of a readmission for each patient; 3) we summed the predicted probabilities from the model for all patients at each hospital and divided the result by the sum of the predicted probabilities if each patient had been treated at an average hospital (i.e., assuming the random effect was zero); and 4) we multiplied this ratio by the overall readmission rate to calculate a standardized readmission rate. Of note, this method “shrinks” hospital readmission rates towards the national mean, particularly for hospitals with low volume. (21)

3M^™ PPR Method

Identification of Index Admissions

We identified the index cohort for the PPR method using the same 1,445,221 VA acute admissions during FY08-10 as we used for the CMS analysis. The PPR software excludes the following cases from the index admission cohort: “error records” (i.e., admission date after death date, mismatched sex, or dates out of order), discharges with a “left against medical advice” code, deaths at discharge, “non-event” hospitalizations (i.e., admission to a non-acute hospital, admission to an acute care hospital for rehabilitation, aftercare of convalescence, or same day transfer to an acute care hospital for non-acute care), and hospitalizations with human immunodeficiency virus (HIV) or metastatic malignancy. Additionally, the PPR software, which uses All-Patient Refined Diagnosis-Related Groups (APR-DRGs) (22) to identify clinically-related readmissions, excludes patients with certain APR-DRGs (i.e., multiple traumas). Finally, hospitalizations with invalid DRG codes (i.e., “error DRGs”) were excluded. (23) To compare our results with the CMS approach, we then identified condition-specific index cohorts for AMI/HF/PNA based on the principal diagnosis of the index admission.

Identification of Readmissions

We obtained the PPRs identified by the 3M PPR software using 30 days as the readmission time interval. The software is designed such that a single index admission may have only one PPR, which is the most common situation, or multiple PPRs (i.e., a Readmission Chain) in order to account for all those readmissions that relate back to a single index admission. The total time period encompassed in a PPR chain may exceed 30 days because the most recent PPR must be within 30 days of the PPR immediately preceding it, not within 30 days of the index admission. (23) The PPR chains do not impact the determination of whether a particular index admission has a readmission or not. However, subsequent readmissions that are part of a PPR chain cannot be new index admissions. The CMS approach, which does not recognize chains, might count a new admission more than 30 days from a previous index admission as a new index admission.

Risk Adjustment

The PPR software indicates APR-DRG and Severity of Illness (SOI) subclass within the APR-DRG. This information can be used to conduct risk-adjusted hospital profiling, although the software itself does not perform these calculations. (12, 23–25)

Calculation of Risk-Adjusted Readmission Rates

We applied the PPR software to generate observed PPR rates for each hospital. We created a VA nationwide normative file that consisted of the average PPR rate for each APR-DRG/SOI subclass based on VA national data. The average PPR rates were then merged with each index discharge (by APR-DRG/SOI subclass) to arrive at a case-level expected PPR outcome. The expected PPR outcomes within a hospital were summed to obtain the number of expected PPR outcomes for the hospital. Each hospital’s risk-adjusted PPR rate was calculated as the number of expected PPR outcomes divided by the number of observed PPR outcomes (O/E ratio).

Detailed comparison of the CMS and PPR methods is shown in Appendix 2 (supplemental digital content 2).

Comparison of the CMS and PPR Risk-Adjusted Readmission Rates

We compared hospitals’ CMS risk-adjusted readmission rates to risk-adjusted PPR rates for each of the three conditions. Specifically, we generated scatter plots and calculated Pearson correlation coefficients (r) to compare the CMS and PPR methods. To avoid potential bias due to low volume, we limited analyses to hospitals with at least 25 index admissions, yielding an exclusion of 27 hospitals for AMI and 1 hospital for HF.

Evaluation of Preventability

To evaluate the role of “preventability” in explaining differences in hospital profiles between these two readmission approaches, we replaced the dependent variable in the CMS HGLM model – Yes/No 30-day all-cause readmission – with the dichotomous PPR outcome Yes/No potentially preventable readmission. We kept everything else the same as in the CMS approach (i.e., condition-specific index admission cohorts, no readmission chains, risk adjustment, and “shrinkage” of hospital rates) in order to isolate the effect of preventability. To examine whether potentially-preventable rates differed as a function of the all-cause rate, we fitted a weighted least squares regression line (using hospital size as weights) with “CMS all-cause rate” as the independent variable and “CMS modified-for-preventability rate” as the dependent variable, and tested whether the slope was equal to one.

Finally, we compared CMS-calculated hospital profiles for public reporting and pay-for-performance using the “CMS all-cause” and “CMS modified-for-preventability” cohorts.

CMS Determination of Hospital Profiles for Reporting

We generated 1,000 bootstrap samples of risk-adjusted CMS readmission rates (repeating for each bootstrap sample steps 1 to 4 of the CMS readmission methods described earlier), which were used to obtain the 95% confidence interval (CI) of each hospital’s risk-adjusted readmission rate. (8–10) High- and low- performing hospitals were determined based on whether the entire CI of the hospital rate was below or above the national observed rate in our VA sample. We compared hospital profiles for the “CMS all-cause” and “CMS modified-for-preventability” cohorts.

CMS Determination of Hospitals Profiles for Payment Adjustment

We identified each hospital’s predicted-to-expected ratio (P/E ratio, i.e., the risk-adjusted predicted readmissions over the risk-adjusted expected readmissions), described in step 3 in the CMS calculation of risk-adjusted readmission rates. The CMS method imposes a payment reduction up to 1% based on P/E ratios for each condition cohort (i.e., aggregate payments for excess readmissions= [sum of base operating DRG payments for AMI × (excess readmission ratio for AMI-1)] + [sum of base operating DRG payments for HF × (excess readmission ratio for HF-1)] +[sum of base operating DRG payments for PN × (excess readmission ratio for PN-1)]). (14) We generated scatter plots to show hospital P/E ratios above and below 1 for the “CMS all-cause” and “CMS modified-for-preventability” cohorts. We also examined how hospital payments might differ due to preventability.

Results

CMS 30-day Risk-Adjusted All-Cause Readmission Rates for AMI/HF/PNA

Our application of the CMS method to identify index admissions in our sample yielded a total of 19,627 AMI, 57,251 HF, and 50,980 PNA hospitalizations (see Figure 1). Characteristics of index admissions are shown in Appendix 1 (supplemental digital content 1). The VA national 30-day all-cause readmission rates were 18.2% for AMI, 22.2% for HF, and 16.7% for PNA. Hospital risk-adjusted readmission rates varied from 14.9 to 23.6% for AMI; 16.5% to 31.0% for HF, and 12.8% to 20.7% for PNA.

Figure 1.

Identification of the CMS and 3M^™ PPR all-condition and condition-specific (AMI, HF, and PNA) index admission and readmission cohorts.

NOTES:

CMS= Centers for Medicare and Medicaid Services

PPR=Potentially Preventable Readmission

AMI=Acute myocardial infarction

HF=Heart failure

PNA=Pneumonia

DRG= Diagnosis Related Groups

APR-DRG=All Patient Refined Diagnosis Related Groups

3M^™ PPR Rates

We identified a total of 1,139,908 index hospitalizations eligible for the PPR software and 159,050 readmissions (see Figure 1). Our final sample for AMI, HF and PNA index cohorts calculated using the PPR method was 19,811, 50,859, and 43,786, respectively. Characteristics of index admissions are shown in Appendix 1 (supplemental digital content 1). The national all-condition PPR rate was 10.8%; facility risk-adjusted all-condition PPR rates ranged from 5.9% to 18.6%. Specific for each cohort, the range of facility risk-adjusted PPR rates was 8.0% to 60.0% for AMI, 9.0% to 31.3% for HF, and 3.9% to 29.2% for PNA.

Comparison of the CMS and PPR Methods

We found moderate correlations between hospitals’ CMS AMI/HF/PN readmission rates and condition-specific cohort PPR rates: correlation coefficients were r=0.42 (n=104) for AMI, r=0.80 (n=130) for HF and r=0.63 (n=131) for PNA (p<0.0001 for all comparisons) (see Figure 2a).

Figure 2.

Figures 2a–b. Scatter plots of hospital risk-adjusted CMS readmission rates and risk-adjusted PPR rates.

Notes:

Figures are generated based on FY08-10 VA acute hospitalizations.

Bubble size is proportional to number of index admissions; only hospitals with >25 index admissions are included (n=104 for AMI, n=130 for HF, and n=131 for PNA)

In Figure 2a, lines indicate median of each rate. In Figure 2b, lines indicate the regression line.

CMS= Centers for Medicare and Medicaid Services

PPR=Potentially Preventable Readmission

AMI=Acute myocardial infarction

HF=Heart failure

PNA=Pneumonia

Evaluation of Preventability

Figure 2b presents the relationship between risk-adjusted “CMS all-cause” versus “CMS modified-for-preventability” rates. We found high statistically significantly correlations between the “CMS all-cause” versus “CMS modified-for-preventability” rates for each cohort (greater than 90% for each comparison). Also, all of the slopes of the regression lines were less than one and statistically significant, indicating that in hospitals with higher frequencies of all-cause readmissions, a higher percentage of those readmissions were potentially preventable.

Our comparison of hospital outliers for public reporting calculated from the risk-adjusted “CMS all-cause” versus “CMS modified-for-preventability” rates found a 34% change among the CMS all-cause outliers (10 of 28 low outliers and 3 of 10 high outliers became average) across the three conditions (see Table 1). Two of the 104 hospitals with a sufficient volume of AMI hospitalizations changed outlier status when the PPR was used to determine preventability. For HF and PNA, eight and six hospitals changed outlier status, respectively. This represents a change in fewer than 7% of hospitals in each condition cohort.

Table 1.

Tables 1a–c. Comparison of “CMS all-cause” vs. “CMS modified-for-preventability” hospital profiles for public reporting on CMS Hospital Compare

Table 1a. Acute Myocardial Infarction (n=104 VA hospitals)
		CMS modified-for-preventability
		Low-Performing	Average	High-Performing	Total
CMS all-cause	Low-performing	4	1	0	5
	Average	0	98	1	99
	High-performing	0	0	0	0
	Total	4	99	1	104

Table 1b. Heart Failure (n=130 VA hospitals)
		CMS modified-for-preventability
		Low-Performing	Average	High-Performing	Total
CMS all-cause	Low-performing	8	4	0	12
	Average	1	108	1	110
	High-performing	0	2	6	8
	Total	9	114	7	130

Table 1c. Pneumonia (n=131 VA hospitals)
		CMS modified-for-preventability
		Low-Performing	Average	High-Performing	Total
CMS all-cause	Low-performing	6	5	0	11
	Average	0	118	0	118
	High-performing	0	1	1	2
	Total	6	124	1	131

NOTES:

CMS= Centers for Medicare and Medicaid Services

NOTES:

CMS= Centers for Medicare and Medicaid Services

NOTES:

CMS= Centers for Medicare and Medicaid Services

To examine the effect of preventability on pay-for-performance, we compared, by cohort, how many hospital P/E ratios changed from below 1 to above 1 or vice versa when the “CMS modified-for-preventability” method was used instead of the “CMS all-cause” method (see Figure 3). Nine hospitals in the AMI cohort (9%) changed their P/E ratio status when preventability was factored into the CMS calculation; 16 hospitals in the HF cohort (12%) and 23 hospitals in the PNA cohort (18%) also changed P/E status (see Figure 3) In our comparison of hospitals for CMS pay-for-performance (i.e., a hospital would be penalized based on whether and how much P/E ratios exceeded 1 in each of the condition cohorts), we found 39 out of 131, or 30% of hospitals changed how much they would be penalized as a result of preventability (see Table 2). Eleven, or 8%, of the hospitals in our sample moved from no financial penalty to some financial penalty when comparing the “CMS all-cause” and “CMS modified-for-preventability” cohorts.

Figure 3.

Hospital predicted-to-expected ratios based on “CMS all-cause” vs. “CMS modified-for-preventability” methods

NOTES:

CMS= Centers for Medicare and Medicaid Services

CMS= Predicted-to-expected ratio based on CMS all-cause readmission model

CMS-P= Predicted-to-expected ratio based on CMS modified-for-preventability readmission model

AMI=Acute myocardial infarction

HF=Heart failure

PNA=Pneumonia

Figures are generated based on FY08-10 VA acute hospitalizations.

Bubble size is proportional to number of index admissions; only hospitals with >25 index admissions are included (n=104 for AMI, n=130 for HF, and n=131 for PNA)

Lines at 1 indicate the threshold for excess readmissions.

Table 2.

Comparison of “CMS all-cause” vs. “CMS modified-for-preventability” hospital profiles for payment adjustment under CMS pay-for-performance

		CMS modified-for-preventability model
		Better in all condition cohorts	Worse in 1 condition	Worse in 2 conditions	Worse in 3 conditions	Total
CMS all-cause model	Better in all conditions	29	5	0	0	34
	Worse in 1 condition	5	24	8	1	38
	Worse in 2 conditions	1	9	28	4	42
	Worse in 3 conditions	0	1	5	11	17
	Total	35	39	41	16	131

NOTES:

CMS= Centers for Medicare and Medicaid Services

The three condition cohorts for readmissions are acute myocardial infarction, heart failure, and pneumonia.

We combined information on predicted-to-expected readmission ratios across the 3 conditions. Not every condition included CMS readmission rates for all 131 hospitals (if volume was under 25, the hospital was not included in the analysis). If a hospital did not have a CMS-generated readmission rate, we considered the hospital profile to be “average.”

Discussion

We analyzed, for AMI/HF/PNA cohorts, three-year aggregated CMS 30-day all-cause readmission rates and, for all conditions, PPR rates for hospitals in the VA. Thirty-day readmission rates based on the CMS method were 18.2% for AMI, 22.2% for HF, and 16.7% for PNA compared to U.S. national rates of 19.8% for AMI, 24.8% for HF, and 18.4% for PNA reported by CMS. (1) As the CMS rates are for Medicare patients over age 64, and our cohort included patients under age 65, it is not surprising that the VA readmission rates were somewhat lower. The VA all-condition PPR rate of 10.8% was nearly identical to the PPR rate the Commonwealth Fund study found for MA in 2009. (19)

When we compared the CMS and PPR readmissions rates for each condition cohort, we found moderate to high correlation. The correlation was strongest for HF (0.80) and weakest for AMI (0.42). For AMI, there was a noticeable number of hospitals (Figure 2a) that had close to median CMS readmission rates but much higher PPR rates, which led to the lower correlation. The reasons for this are not clear; future research should consider characteristics of these hospitals and AMI patients at these hospitals in more detail. However, when we kept everything constant except whether or not a readmission was potentially preventable, we found a strong correlation (over 0.90) between the CMS and PPR readmission rates for all three conditions.

We found that the two readmission approaches, when applied to the AMI/HF/PN cohorts, identified somewhat different high- or low-performing hospitals. When we held everything constant except preventability, we found that fewer than 7% of the hospitals in each cohort would have had different outlier status for public reporting, but 30% of the hospitals would have had different payment penalties. These differences give rise to the question “which method is better?” Our results do not provide much insight into the answer to this question because no study has yet determined whether potentially preventable readmissions as identified by the PPR method are in fact any more preventable than other readmissions. Until such time, we believe that the CMS measures are better suited for hospital profiling because they are standardized, publically accessible, and have been endorsed by National Quality Forum. (13) For internal hospital quality improvement it seems reasonable to concentrate on those cases most likely to be preventable (i.e., the ones identified by the PPR approach).

The discrepancies in hospital profiling for reporting and pay-for-performance between the CMS and PPR methods have important policy implications both within and outside the VA. If VA hospitals faced financial incentives related to 30-day readmission rates, six hospitals would no longer receive any penalty and five hospitals would receive some penalty if preventability were factored into the CMS calculation. The implications of potentially contradictory hospital readmission profiles on quality improvement initiatives and subsequent patient care are not well understood.

Although we conducted our analysis in the VA, we believe the results are generalizable to the private sector. Poor generalizability of VA studies are typically due to the unique VA patient population; because the discrepancies in CMS and PPR-generated hospital profiles are largely caused by methodological differences rather than hospital case-mix, we believe our findings are meaningful to private sector hospitals. A further strength of our study is the use of a national hospital sample, which mitigates potential bias from regional variation in patient care. Finally, though we cannot state definitively whether the PPRs measure truly preventable readmissions, we have shown, particularly for pay-for-performance as being implemented by CMS, inclusion of preventability in the definition of readmissions has a significant impact on which hospitals will suffer payment penalties.

Conclusion

Despite the uncertainty over which 30-day readmission method is superior in evaluating hospital quality, we confirmed that there are discrepancies in CMS- and PPR-generated hospital profiles. This may largely be driven by the well-known differences in the methodologies, including the PPR’s preventability component. Future research should focus on validating and enhancing these two methods with respect to hospital profiling so that hospitals and policy makers can use them to monitor trends, develop effective quality improvement initiatives, and reduce preventable readmissions.

Supplementary Material

Appendix 1

Appendix 1. Characteristics of CMS AMI/HF/PN index admissions and PPR all-condition and AMI/HF/PN index admissions

Appendix 2

APPENDIX 2. Comparison of CMS and 3M PPR Methods