Expanding the Frontier of Outcomes Measurement for Public Reporting

The decade from 2000 onward was notable for many changes in healthcare, but among the most significant was the Centers for Medicare and Medicaid Services’ (CMS) effort to develop a large public reporting program. Hospital performance profiling at the national level had not figured prominently since 1986, when the Health Care Financing Administration (HCFA, now CMS) publicly reported hospital-specific mortality rates for numerous medical and surgical diagnoses. CMS began its 21^st Century efforts by measuring and reporting processes of high-quality care for acute myocardial infarction, heart failure, pneumonia, and general surgery, such as prescription of aspirin at discharge for patients hospitalized for acute myocardial infarction and timely delivery of antibiotics for patients admitted for pneumonia.

In July 2007, CMS began publicly reporting outcomes for the first time: hospital-specific risk-standardized mortality rates for patients hospitalized for acute myocardial infarction or heart failure. A year later, risk-standardized mortality rates were reported for patients hospitalized for pneumonia as well. Then in July 2009, CMS added hospital-specific risk-standardized readmission rates for patients hospitalized for these same three conditions. Since that time, CMS has continued to develop outcome measures for additional conditions and the recently enacted Patient Protection and Affordable Care Act (PPACA) contains several requirements for CMS to expand outcome measurement and public reporting, including the development of additional measures of hospital quality as well as measures of primary care and preventive care quality.

Public reporting is considered one of many critical strategies among efforts to improve healthcare quality, as it leverages three key change pathways.^{1, 2} First, public reporting can promote informed choices by patients as well as informed referrals by providers and hospitals are expected to focus on improving quality in order to increase hospital market share. Second, public reporting can affect the public image or reputation of a clinician, hospital, or other medical care organization, and hospital concern for its public image can motivate quality improvement. Third, public reporting can provide objective feedback that fuels professional desire to improve care and improve quality. However, for public reporting to be effective through these pathways, the measures must be considered important, scientifically sound, usable, and feasible.³

Working at the Yale-New Haven Hospital Center for Outcomes Research and Evaluation, we and our colleagues continue to play a role in the development and maintenance of the outcome measures that are used by CMS for public reporting. Each of the outcome measures used by CMS for public reporting have been designed with the above criteria in mind and have been endorsed by the National Quality Forum.^{4, 5} The measures currently reported are estimated using administrative claims data, calculated from two-level (patient and hospital) hierarchical logistic regression models that account for the clustering (non-independence) of patients within the same hospital and that also account for patient-specific information on age and sex as well as a number of clinical characteristics identified from the past medical history using diagnostic codes. In order to ensure that the process of measuring outcomes for hospitals was feasible in today's health care system, the use of administrative data was a necessity. However, to test their scientific soundness during measure development, hospital risk-standardized rates were calculated and compared when using administrative claims data and when using data obtained from medical records. The administrative claims rates have consistently been found to be good surrogates for rate estimates from medical record data.^{6, 7}

In this month's issue of Circulation: Cardiovascular Quality and Outcomes, Hammill and colleagues consider alternative sources of data for performance measurement, a key issue looming on the horizon for the field. As electronic medical records are increasingly common and sophisticated, the incremental value of using medical record data for measurement needs to be evaluated. At some time in the future, medical record data may be as easily accessible as administrative claims data is today. We need to ask, will we then be able to improve performance measurement? Intuitively, it would seem so and Hammill begins to shed some light on this issue.

Using the 2004-2006 Get With The Guidelines-Heart Failure registry, which includes upwards of 300 hospitals, and linking to Medicare administrative claims data for fee-for-service beneficiaries, Hammill and colleagues examined the incremental value of adding medical record data to the claims-based estimates of hospital performance on outcome measures of mortality and readmission for patients hospitalized with heart failure.⁸ Specifically, they examined the impact of additionally including information on ejection fraction, heart rate, serum hemoglobin, serum creatinine, serum sodium, systolic blood pressure, and weight from the medical record when risk-adjusting the administrative claims-based performance measures. The authors then ranked hospitals in quintiles by their predicted performance, categorizing hospitals into the top 20%, middle 60%, or bottom 20%, using both the claims-based and the claims plus medical record estimates and plotted the agreement between the rankings. In addition, they reported differences between the two estimates in which hospitals were determined to be outliers.

Interestingly, for readmission risk, Hammill and colleagues found that the claims plus medical record estimates operated no better than the claims alone estimates and had little impact on performance ranking. However, for mortality risk, the claims plus medical record estimates had greater discrimination when compared with the claims alone estimates and affected the performance ranking categorization of 10% of hospitals.

This work done by Hammill and colleagues informs current and future policy decisions. First, the study provides some reassurance with regards to the data source currently used by CMS for the measurement and public reporting of readmission risk after hospitalization for heart failure, as the claims plus medical record estimates were no better than the claims alone estimates. However, it is important to note that the study examined only a limited sample of hospitals and only select clinical variables were tested. Second, although the claims plus medical record estimate improved discrimination and affected performance rankings for mortality measurement, the categorization of 90% of hospitals did not change when including additional medical record information for risk adjustment. Moreover, this effect is likely to be less substantial when using CMS's definition of outliers. CMS identifies hospitals as being better or worse outliers only if their rates differ significantly from the national average with a high level of certainty (using a 95% interval estimate), whereas Hammill identified outlier hospitals as those ranked among the top 20% or bottom 20%. At the margin, medical record data may slightly improve the predictive accuracy of estimates measuring hospital mortality. However, there are other inherent limitations to using medical record data, including extraction costs, precision of physiologic measurements, and management of missing information, which can be substantial. Nevertheless, at this time, this data is not widely available. We must continue to examine whether medical record data can be used to improve outcomes measurement as it becomes more readily available, weighing such improvements against the costs of obtaining reliable data.

This month's issue also contains a second article related to the expanding frontier of outcomes measurement as proscribed by recently enacted legislation. Section 3025 of the PPACA requires CMS to initiate a Hospital Readmissions Reduction Program, such that hospitals with ‘excessive’ 30 day readmissions will incur financial penalties in the form of reduced payments beginning in the 2013 fiscal year. Joynt and Jha explore the critical question of whether readmission rates for patients hospitalized for heart failure at resource-poor hospitals are higher than average. If resource-poor hospitals demonstrate worse performance, they may be at increased risk for having their payments reduced under such a program and potentially be poised to experience a downward spiral that leaves these hospitals even more resource-poor.

Using 2006-2007 Medicare administrative data and data from the American Hospital Association, Joynt and Jha examined the association between hospital characteristics and 30 day readmission rates among patients discharged after hospitalization for heart failure.⁹ They defined several hospital attributes as being characteristic of a resource-poor hospital, including public ownership, location in poorer counties, smaller volumes, lower levels of nurse staffing, and lacking capacity to provide advanced cardiac care such as percutaneous coronary interventions or coronary artery bypass graft surgery. The authors then calculated the hospital readmission rates and ranked hospitals by their predicted performance, categorizing those ranked in the highest quartile as “poor performers.” Of note, Joynt and Jha do not use the two-level hierarchical logistic regression models employed by CMS for measurement and public reporting, using instead generalized estimating equations.

Joynt and Jha found that all five attributes characteristic of either a financially or clinically resource-poor hospital were independently associated with higher readmission rates in multivariable models and that four of five, with the exception of county income, were independently associated with greater odds of being categorized as a “poor performer.”

The decision by Joynt and Jha to use a statistical approach that differs from the one currently used by CMS for measuring readmission rates somewhat limits the interpretation of their study. The authors justify the use of general estimating equations by stating that their objective was to “identify differences in performance,” contending that CMS’ use of hierarchical logistic regression modeling “reduces variability in the data, particularly for small hospitals.” This critique of the CMS statistical approach deserves comment. When developing outcome measures for performance measurement and public reporting, there should not be prejudgment as to the amount of variation in quality or performance that exists, with selection of a statistical approach to accentuate variation. Rather, the goal should be to report the true variation in quality among hospitals. To accomplish this, outcomes measures should accurately estimate the hospital-specific contribution to patient outcomes after adjusting for patient factors, which requires distinguishing between-hospital and within-hospital variation, as is done through the CMS approach.

Furthermore, the purpose of Joynt and Jha's investigation was to determine whether resource-poor hospitals would be more likely to be identified as worse performers and thus subject to financial penalties in a hospital payment reduction program focused on readmission rates. Ultimately, however, the differences among hospitals reported in this study may not be practically meaningful, precisely because they have chosen a differing statistical approach. The authors recognize this contradiction by simultaneously raising concerns about resource-poor hospitals being worse performers on the one hand, while on the other acknowledging that differences between resource-poor hospitals and others are substantially smaller when using the CMS' statistical approach.

Nevertheless, we share Joynt and Jha's interest in continued examination of the potential consequences of performance measurement, public reporting, and payment reduction programs on resource-poor hospitals. Others have similarly raised concerns as to whether public reporting and pay for performance programs are likely to widen disparities in care,^{10, 11} as opposed to improving healthcare quality. Our own work in this area, using the CMS measures and focused on several different hospital characteristics, including safety-net status, bed size, ownership, teaching status, and capacity to provide advanced cardiac care, demonstrated substantial variation within hospital characteristic categories and a similar range of performance for hospitals across these categories.^{12, 13} Moreover, there were substantial numbers of resource-poor hospitals among the best performers, despite limited financial and clinical resources.¹² These issues warrant continued research and surveillance.

The advantages of using a well-designed risk-standardized statistical approach to measuring readmission rates for payment incentive programs should not be overlooked. The current measures are scientifically sound, well-validated, feasible and usable. They reveal important variation among hospitals that programs can be targeted to improve. In general, public reporting and outcomes measurement, tied to payment incentives, are expected to continue the rapid growth we have seen through the first 10 years of the 21^st century. As a strategy, government and other insurance payors are embracing these programs, anticipating they will improve quality while limiting costs. But the field of performance measurement is growing just as quickly. The work performed by both Hammill and colleagues and Joynt and Jha is critical to its advancement, exploring how measurement can be improved while monitoring for any potential unintended consequences of the programs.