Abstract
Background
Population-based measures of admissions among patients with chronic conditions are important quality indicators of Accountable Care Organizations (ACOs), yet there are challenges in developing measures that enable fair comparisons among providers.
Methods
Based on consensus standards for outcome measure development and with expert and stakeholder input on methods decisions, we developed and tested two models of risk-standardized acute admission rates (RSAARs) for patients with diabetes and heart failure using 2010–2012 Medicare claims data. Model performance was assessed with deviance R-squared; score reliability was tested with intraclass correlation coefficient. We estimated RSAARs for 114 Shared Savings Program ACOs in 2012 and we assigned ACOs to 3 performance categories: no different, worse than, and better than the national rate.
Results
The diabetes and heart failure cohorts included 6.5 and 2.6 million Medicare fee-for-service (FFS) beneficiaries aged ≥65 years, respectively. Risk-adjustment variables were age, comorbidities and condition-specific severity variables, but not socioeconomic status or other contextual factors. We selected hierarchical negative binomial models with the outcome of acute, unplanned hospital admissions per 100 person-years. For the diabetes and heart failure measures respectively, the models accounted for 22% and 12% of the deviance in outcomes and score reliability was 0.89 and 0.77. For the diabetes measure, 51 (44.7%) ACOs were no different, 45 (39.5%) were better, and 18 (15.8%) were worse than the national rate. The distribution of performance for the heart failure measure was: 61 (53.5%);,37 (32.5%) and 16 (14.0%), respectively.
Conclusion
Measures of RSAARs for patients with diabetes and heart failure meet criteria for scientific soundness and reveal important variation in quality across ACOs.
Introduction
Accountable Care Organizations (ACOs) are designed to improve the quality of health care and the health of the populations they serve while limiting the growth in healthcare costs.1 In the Centers for Medicare & Medicaid Services’ (CMS’s) Shared Savings Program, providers voluntarily come together to provide efficient, coordinated care for a population of Medicare Fee-for-Service (FFS) beneficiaries.2 Outcome measures are an important component of such programs, serving to assess whether these new provider arrangements are benefiting patients.. Hence, CMS has supported the development and use of new quality outcome measures that focus on population-based admission rates among patients with chronic conditions in the Medicare Shared Savings Program.
Acute hospital admissions are important health outcomes for patients with chronic conditions and serve as useful indicators of the quality of chronic disease management and care coordination.3–8 Patients with chronic conditions are susceptible to exacerbations of, and complications from, their underlying disease, and are vulnerable to other acute illnesses, some of which may be averted with high-quality care.3,9 To assess ACOs’ ability to manage complex patients with chronic conditions, we developed two outcome measures of acute, unplanned admissions for use in the Shared Savings Program. These measures focus on patients with diabetes and patients with heart failure because these patients are at high risk for hospitalization. Moreover, diabetes and heart failure affect 18% and 14% of Medicare beneficiaries and these patients account for 32% and 43% of Medicare spending, respectively.10
CMS recently added these measures to the ACO pay-for-reporting and pay-for-performance measure set.11 Other existing ACO outcomes measures are limited. For example, several Agency for Healthcare Research and Quality (AHRQ) Prevention Quality Indicators (PQIs) have been adapted for the ACO program. These PQIs are based on admission rates but focus on a narrow outcome (exacerbations of the condition of interest), lack risk adjustment for comorbidities and disease severity, and include planned (as well as unplanned) admissions.
The development of these measures required alignment of methodological decisions with the Shared Savings Program structure and goals.2 Specifically, the measures needed to define the outcomes that best reflect quality, adequately risk adjust for case-mix differences across providers, and account for the nested structure of patients within ACOs.12,13 In addition, in designing the risk-adjustment strategy, these measures required decisions about which factors may influence admission rates but differ across ACOs; we also needed to consider whether ACOs can be held accountable for such factors in the complex ambulatory care environment. For example, many clinical, behavioral, social, and community factors may affect the risk for admission. While some of these may be outside of ACOs’ scope of influence, the ACO program structure is designed to incentivize providers to think broadly about their ability to improve the health of the people they serve. For all of these methodologic decisions, we sought input from experts and the public, and adhered to guidelines for scientific soundness set forth by the National Quality Forum.
In this paper we present the methodologic approach to to measure development, describe the rationale for key methods decisions, and report model performance and measure score reliability testing. In addition, we report the measure scores (RSAARs) for Shared Savings Program ACOs existing in 2012 and evaluate variation in ACO performance relative to the national rate of admissions for all Medicare FFS beneficiaries with diabetes and with heart failure.
Methods
To develop these measures, we assembled a multidisciplinary team of clinicians, health services researchers, and biostatisticians. In accordance with CMS’s standardized, transparent process, we convened a national technical expert panel (TEP) comprised of patients, health industry representatives, researchers, and healthcare providers with expertise in diabetes, heart failure, and geriatrics. We also held a public comment period soliciting stakeholder input on the measure methodology, and refined the measure in response to comments. We tested the measures against the National Quality Forum’s (NQF’s) criteria for scientific soundness and importance12,13 including testing the risk-adjustment model properties and evaluating the measure score variation across Shared Savings Program ACOs.
Data Sources
To develop each measure, we assembled Medicare Fee-for-Service (FFS) claims data from 2010–2012 linked using unique patient identifiers. To define the cohorts and identify each patient’s risk factors, we used 2010–2011 Medicare Part A and Part B claims from the Chronic Conditions Data Warehouse which consists of 100% of FFS patients. We used the 2012 Medicare Provider Analysis and Review 100% FFS dataset of Medicare Part A claims to assess the outcome of admissions. To determine Medicare FFS enrollment, demographic, and death information, we used Medicare denominator files. CMS provided a data file identifying the 114 Shared Savings Program ACOs that participated in 2012 and beneficiary assignment.
For measure development and testing, we randomly split the full sample into a development and a validation sample for each of the two measures. In addition, for measure score reliability testing, we randomly split the full sample into two reliability testing samples by randomly splitting each ACO’s patients in half and then randomly splitting all non-ACO patients in half.
Measure Cohorts
The target populations for these measures are Medicare FFS patients aged ≥65 years with diabetes or heart failure. We sought to define cohorts inclusive of patients at all stages of diabetes and heart failure while ensuring the measures can fairly balance differences in patient mix across ACOs.
Patients with diabetes had at least one inpatient or two outpatient claims for diabetes (in any position on the claim) within the two years prior to the measurement period (Appendix Table 1). Patients with heart failure had at least one hospital claim with a principal diagnosis code for heart failure or two claims (inpatient or outpatient) with codes for heart failure in any position within the two years prior to the measurement period (Appendix Table 2). In order to adequately assess covariates for risk adjustment, we required continuous enrollment in Medicare Parts A and B during the year prior to the measurement period (i.e., year 2011).
For both measures, we excluded patients without continuous enrollment in Medicare Part A during the measurement period (i.e., 2012) in order to assess the outcome. For the heart failure cohort only, we excluded patients with left ventricular assist devices (LVADs) because they are high-risk patients clustered among a few ACOs.
Outcome
The outcome for both measures was the number of acute, unplanned admissions per 100 person-years at risk for hospitalization. Persons were considered at risk for admission if they were alive, enrolled in FFS Medicare, and not currently admitted. Admissions were defined as any inpatient admission to a short-term acute care hospital for any cause during the measurement year, unless an admission was identified as “planned.” To identify planned admissions, we adapted a planned admission algorithm developed for CMS’s hospital readmission measures with two adaptations: we removed cardiac catheterization and amputation of lower extremity from the planned procedure list as these procedures are frequently unplanned admissions among ambulatory patients with diabetes and heart failure.14 The outcome was measured over one year, consistent with the structure of the ACO program.2
Covariates for Risk-Adjustment
In designing the risk-adjusted models, we considered several factors that may affect performance on the measures, but not be an indicator of ACO quality. They included:
Selection of clinical variables that increase risk of admission. We assessed risk factors in the year prior to the measurement of the outcome. Some of these risk factors may reflect the quality of chronic disease management over many years, presumably before the ACO assumed care of these patients. Candidate risk variables were selected from 189 diagnostic condition groups included in CMS’s Hierarchical Condition Category clinical classification system.15 Details of candidate variable selection are included in the Appendix
Accounting for patients’ disease severity, which may vary across ACOs. For the diabetes measure we used a diabetes complications severity index that has been validated in claims data.16,17 The index takes on values from 0 to 7, according to the number of complications present (i.e., retinopathy, nephropathy, neuropathy, cerebrovascular, cardiovascular, peripheral vascular disease, and metabolic complications). For the heart failure measure, we considered a history of implantable cardiac defibrillator (ICD), cardiac resynchronization therapy (CRT), or permanent pacemaker (PPM) to identify patients with more advanced disease. While PPM does not necessarily reflect advanced heart failure, the unadjusted rates of admission were similar for ICD, CRT, and PPM.
Consideration of sociodemographic factors that may impact the rate of admission, including age, sex, race, and socioeconomic status (SES). With the exception of age, we did not adjust for other demographic variables. Differences in the risk for hospitalization among patients of different sociodemographic characteristics may represent disparities in the delivery and quality of care, which we do not want to obscure by including these variables in the risk-adjustment model. Nevertheless, we recognize that there are discordant data with respect to outcomes by sex.18–20 Additionally, we appreciate the concern that outcome measures may be unfair to ACOs caring for disproportionately low socioeconomic populations.21,22 To address these concerns, we tested the impact on measure scores of including sex and SES in the risk-adjustment models.
Consideration of other aspects of population health, such as healthy behaviors and community resources. We did not include such factors in the risk-adjustment models; as part of their mission, ACOs are encouraged to influence health behaviors, and to develop strategic partnerships with community-based organizations and businesses, in order to improve population health and to reduce the risk of admission.
We used the Akaike Information Criterion (AIC) values to select the best-fitting model using the fewest candidate variables; AIC is commonly used in variable selection for negative binomial models (which use count data, such as a count of the number of admissions) to account for overdispersion (whereby data vary more than expected).23
Statistical Model
The RSAAR for each ACO was calculated as the number of predicted to the number of expected admissions per 100 person-years, multiplied by the national rate of admissions among all Medicare FFS patients with diabetes or heart failure. The measure uses a two-step statistical model that accounts for the clustering of patients within ACOs and accommodates the varying sizes of different ACOs. Among models appropriate for use with count data, in this case the number of admissions, we selected a negative binomial model form because it best accounted for overdispersion within the data (more variance in the data than predicted).
The first step of the model (patient-level) adjusts for patient risk factors. The relationship between patient risk factors and the outcome of admission is determined based on all patients with diabetes or all patients with heart failure. Since the effects that risk factors exert on the number of admissions are estimated based on data from all patients in the nation, irrespective of whether they are cared for by an ACO, the expected number of admissions for each ACO is determined by all providers and patients nationally. Hence, ACO performance is assessed relative to patients cared for by ACO and non-ACO providers.
The second step of the model is a hierarchical model. This model estimates an ACO-level random-intercept term that reflects the ACO’s contribution to admission risk. This ACO-level random effects term is based on the ACO’s actual admission rate, the performance of other providers, and the ACO sample size. The model takes into account ACO sample size only insofar that estimates of the ACO-level random effects term for ACOs of small size are less certain. Therefore, ACOs with few cases are assumed to perform more like average providers.
Analyses
To assess performance of the patient-level model for each measure, we computed two summary statistics: (1) goodness-of-fit statistic (deviance R-squared) and (2) overfitting indices. We randomly split each measure cohort into a development and validation set and calculated deviance R-squared using the model deviance residual24 and over-fitting indices.
To assess the effect of sex and low SES status (defined as Medicare-Medicaid dual-eligibility) on model performance, we compared the correlation between measure scores with and without each variable in the models using the Spearman correlation coefficient.
To assess the reliability of ACO scores (that is, to test the measure’s ability to assess ACO quality consistently with repeated measures), we randomly sampled half of the patients from each ACO and half of the patients who were not in ACOs, and compared measure scores between the two samples. Thus, each ACO was measured twice, but each measurement was made using an entirely distinct set of patients. As a metric of agreement we calculated the intraclass correlation coefficient (ICC).25,26
For each ACO, we calculated an interval estimate (IE) using bootstrapping methodology. Using 95% IEs, we assigned ACOs to one of three performance categories: “better than the national rate,” “no different than the national rate,” and “worse than the national rate,” with “better than” and “worse than” falling below and above the 95% IEs, respectively.
Statistical analyses were performed using SAS version 9.3 (SAS Institute Inc., Cary, NC).
Results
Cohorts
For the diabetes measure, after exclusion of patients without continuous enrollment in Medicare Part A in 2012 (n=225,314), the cohort included 6,521,462 patients. The majority of patients were female (54.7%); average age was 76.4 years; 341,193 (5.2%) were assigned to one of 114 Shared Savings Program ACOs (Table 1A).
Table 1.
| Medicare Beneficiaries With Diabetes Mellitus
|
P | |||
|---|---|---|---|---|
| Total (N = 6,521,462) | Non-ACO (N = 6,180,269) | ACO (N = 341,193) | ||
| ACO assignment in 2012 | ||||
| Proportion assigned to Shared Savings Program ACOs (%) | 5.2 | |||
| Sociodemographic factors | ||||
| Age [mean (SD)] | 76.4 (7.2) | 76.4 (7.2) | 75.9 (7.0) | <0.0001 |
| Sex, male (%) | 45.3 | 45.0 | 50.7 | <0.0001 |
| Race (%) | <0.0001 | |||
| White | 81.5 | 81.4 | 83.6 | |
| Black | 11.0 | 11.1 | 10.3 | |
| Asian | 2.4 | 2.4 | 2.3 | |
| Hispanic | 2.4 | 2.4 | 1.7 | |
| North American native | 0.6 | 0.6 | 0.1 | |
| Other | 1.8 | 1.8 | 1.9 | |
| Unknown | 0.2 | 0.2 | 0.1 | |
| Dual eligible (%) | 19.5 | 19.8 | 14.0 | <0.0001 |
| Clinical factors (%) | ||||
| High-risk cardiovascular factors | 16.9 | 16.9 | 16.6 | <0.0001 |
| Low-risk cardiovascular factors | 57.0 | 57.0 | 57.3 | <0.0001 |
| Arrhythmia | 29.1 | 29.1 | 29.1 | 0.7234 |
| Structural heart disease | 18.4 | 18.4 | 19.0 | <0.0001 |
| Advanced cancer | 5.6 | 5.6 | 5.9 | <0.0001 |
| Dementia | 14.2 | 14.4 | 12.2 | <0.0001 |
| Heart failure | 23.0 | 23.1 | 21.8 | <0.0001 |
| Dialysis | 1.5 | 1.5 | 1.4 | <0.0001 |
| Disability/frailty | 13.5 | 13.6 | 12.4 | <0.0001 |
| Gastrointestinal/genitourinary disease | 23.3 | 23.3 | 23.7 | <0.0001 |
| Hematological disorders | 8.5 | 8.5 | 8.6 | 0.0031 |
| Infectious/immune disorders | 3.4 | 3.4 | 3.4 | 0.0929 |
| Kidney disease | 22.6 | 22.6 | 22.6 | 0.6568 |
| Liver disease | 1.7 | 1.7 | 1.7 | 0.0201 |
| Neurological disorders | 26.1 | 26.1 | 25.5 | <0.0001 |
| Psychiatric disease/substance abuse | 26.7 | 26.8 | 25.2 | <0.0001 |
| Pulmonary disease | 37.1 | 37.2 | 36.2 | <0.0001 |
| Other advanced organ failure | 7.6 | 47.7 | 6.9 | <0.0001 |
| Iron deficiency anemia | 36.8 | 36.8 | 36.7 | 0.4733 |
| Major organ transplant | 0.2 | 0.2 | 0.3 | 0.0002 |
| Other organ transplant | 0.6 | 0.6 | 0.6 | 0.4587 |
| Hip fracture/major fracture | 3.4 | 3.4 | 3.1 | <0.0001 |
| No. diabetes complications [mean (SD)] | 1.41 | 1.67 (1.41) | 1.70 (1.41) | <0.0001 |
ACO indicates Accountable Care Organizations.
For the heart failure measure, after exclusion of patients without continuous enrollment in Medicare Part A in 2012 and of patients with an LVAD (n=66,900 and 1,048, respectively), the cohort included 2,581,892 patients. The majority of patients were female (56.9%); average age was 80.4 years; 123,626 (4.8%) were assigned to one of 114 Shared Savings Program ACOs (Table 1B).
There were no differences in the demographic or clinical factors for the split development and validation samples when compared with the overall samples for the two cohorts (data not shown).
Candidate and Final Variables
For each of the measures, the best combination of variables based on AIC was 23 of the 24 candidate variables [see Appendix Tables 3 and 4], all of which were significantly associated with the outcome (p<0.05). Table 2 shows final model variables and their rate ratios for each measure.
Table 2.
| Medicare Beneficiaries With Heart Failure
|
P | |||
|---|---|---|---|---|
| Total (N = 2,581,892) | Non-ACO (N = 2,458,266) | ACO (N = 123,626) | ||
| ACO assignment in 2012 | ||||
| Proportion assigned to Medicare Shared Savings Program ACOs (%) | 4.8 | |||
| Sociodemographic factors | ||||
| Age [mean (SD)] | 80.4 (8.0) | 80.5 (8.0) | 79.8 (7.8) | <0.0001 |
| Sex, male (%) | 43.1 | 42.7 | 50.8 | <0.0001 |
| Race (%) | <0.0001 | |||
| White | 85.5 | 85.4 | 87.2 | |
| Black | 9.7 | 9.7 | 9.2 | |
| Asian | 1.4 | 1.4 | 1.1 | |
| Hispanic | 1.9 | 1.9 | 1.3 | |
| North American native | 0.4 | 0.5 | 0.1 | |
| Other | 1.0 | 1.0 | 1.0 | |
| Unknown | 0.1 | 0.1 | 0.1 | |
| Dual eligible (%) | 23.6 | 23.9 | 16.9 | <0.0001 |
| Clinical factors (%) | ||||
| High-risk cardiovascular factors | 32.5 | 32.5 | 32.8 | 0.0243 |
| Low-risk cardiovascular factors | 84.4 | 84.4 | 85.4 | <0.0001 |
| Arrhythmia | 62.7 | 62.6 | 74.4 | <0.0001 |
| Structural heart disease | 39.7 | 39.6 | 41.5 | <0.0001 |
| Advanced cancer | 7.3 | 7.3 | 7.8 | <0.0001 |
| Dementia | 25.7 | 25.9 | 22.4 | <0.0001 |
| Diabetes with complications | 51.7 | 51.6 | 52.6 | <0.0001 |
| Dialysis | 3.0 | 3.0 | 3.1 | 0.0091 |
| Disability/frailty | 24.2 | 24.3 | 22.9 | <0.0001 |
| Gastrointestinal/genitourinary disease | 32.2 | 32.2 | 32.6 | 0.0016 |
| Hematological disorders | 16.0 | 16.0 | 16.7 | <0.0001 |
| Infectious/immune disorders | 6.1 | 6.1 | 6.4 | 0.0002 |
| Kidney disease | 38.2 | 38.2 | 39.1 | <0.0001 |
| Liver disease | 2.4 | 2.4 | 2.3 | 0.1313 |
| Neurological disorders | 45.8 | 45.8 | 45.1 | <0.0001 |
| Psychiatric disease/substance abuse | 38.7 | 38.8 | 36.5 | <0.0001 |
| Pulmonary disease | 60.4 | 60.4 | 59.3 | <0.0001 |
| Other advanced organ failure | 21.2 | 21.2 | 20.3 | <0.0001 |
| CRT/ICD/pacemaker | 21.9 | 21.8 | 23.7 | <0.0001 |
| Iron deficiency anemia | 54.1 | 54.1 | 54.0 | 0.4755 |
| Major organ transplant | 0.3 | 0.3 | 0.4 | 0.0014 |
| Other organ transplant | 0.8 | 0.8 | 0.8 | 0.3766 |
ACO indicates Accountable Care Organizations; CRT, cardiac resynchronization therapy; ICD, implantable cardiac defibrillator.
Model Performance
For the diabetes measure, the deviance R-squared in the development and validation samples were 0.217 and 0.218, respectively, and for the heart failure measure were 0.122 and 0.123. These indicate that the models explain ~22% and ~12% of the variation in admissions across patients with diabetes and heart failure, respectively. Overfitting indices were (0.0017, 1.0031) and (−0.0020, 1.0002) for the diabetes and heart failure measures, respectively; the overfitting index of γ0 close to 0 and γ1 close to 1 indicates good calibration of the models. Additionally, the plots of observed and predicted probabilities for each number of hospital admissions (0, 1, 2, …, 10) across four risk groups show that the models perform well across a broad range of risk categories (see Appendix Figure 1 and Figure 2).
Figure 2.
Performance of Shared Savings Program ACOs on the diabetes and heart failure measures. Percentage of ACOs is on the y-axis. Three categories of performance are on the x-axis: better than national rate, no different than national rate, and worse than national rate. ACO indicates Accountable Care Organizations.
ACO-Level Measure Score
Diabetes Measure
In 2012, there were 2,940,537 hospital admissions, with 353,192 (12.0%) classified as planned admissions, resulting in a total of 2,587,345 acute, unplanned admissions.
The crude national Medicare FFS rate of acute, unplanned admissions was 41.4 per 100 person-years. Among ACOs, the mean RSAAR was 39.6 admissions per 100 person-years (standard deviation = 7.3), whereas the median was 39.1 (interquartile range [IQR] 34.8 to 43.9). The minimum, 5th percentile, 95th percentile and maximum RSAAR was 23.9, 28.6, 53.0 and 68.1 admissions per 100 person-years, respectively (Figure 1A).
Figure 1.
A, Distribution RSAARs across ACOs for the diabetes measure. Number of ACOs is on the y-axis, RSAAR is on the x-axis. The grey vertical line represents median RSAAR. B, Distribution of RSAARs across ACOs for the heart failure measure. Number of ACOs is on the y-axis, RSAAR is on the x-axis. The grey vertical line represents median RSAAR. ACO indicates Accountable Care Organizations; RSAARs, risk-standardized acute admission rates.
51 ACOs (44.7%) had RSAARs that were “no different” from the national Medicare FFS rate among patients with diabetes. An additional 45 ACOs (39.5%) had RSAAR scores “better than the national rate” and 18 ACOs (15.8%) “worse than the national rate” (Figure 2).
Heart Failure Measure
In 2012, there were 2,123,190 hospital admissions, with 145,443 (6.9%) classified as planned admissions, resulting in a total of 1,977,747 acute, unplanned admissions.
The crude national Medicare FFS rate of acute, unplanned admissions was 85.5 per 100 person-years. Among ACOs, the mean RSAAR was 81.9 per 100 person-years (standard deviation = 11.6), whereas the median was 81.5 ([IQR] 73.6 to 88.8). The minimum, 5th percentile, 95th percentile and maximum RSAAR was 53.7, 64.6, 101.7 and 120.7 admissions per 100 person-years, respectively (Figure 1B).
61 ACOs (53.5%) had RSAARs that were “no different” from the U.S. national Medicare FFS rate among patients with heart failure. An additional 37 ACOs (32.5%) had RSAAR scores “better than the national rate” and 16 (14.0%) “worse than the national rate” (Figure 2).
Measure Score Reliability
The ICC was 0.89 for the diabetes measure and 0.77 for the heart failure measure, which according to the conventional interpretation is considered “almost perfect agreement” between two datasets.26 Thus, the measures reliably assessed ACO quality when calculated based on two separate datasets.
Sensitivity Analyses
Comparing the RSAAR with and without sex included in the models resulted in a high degree of correlation in both the diabetes and heart failure measures (Spearman correlation coefficient = 0.999 and 0.999, respectively). Similarly, comparison of RSAARs with and without adjustment for Medicaid dual-eligibility status were highly correlated for both measures (Spearman correlation coefficient = 0.976 [Figure 3A] and 0.991, respectively [Figure 3B]). Hence, the measure scores were very similar whether or not sex or Medicaid dual eligibility was included in the models.
Figure 3.
A, Plot of RSAARs with and without adjustment for Medicaid dual-eligibility status among patients with diabetes. RSAAR adjusted for dual-eligibility status is on the y-axis. RSAAR (unadjusted) in on the x-axis. B, Plot of RSAARs with and without adjustment for Medicaid dual-eligibility status among patients with heart failure. RSAAR adjusted for dual-eligibility status is on the y-axis. RSAAR (unadjusted) in on the x-axis. RSAARs, risk-standardized acute admission rates.
Discussion
We developed two population-based, risk-standardized measures of acute hospital admissions for patients with diabetes and patients with heart failure. The measures account for differences in clinical risk profiles of patients cared for by different ACOs (case-mix) and can be used to assess ACO performance. Consistent with the National Quality Forum’s (NQF’s) criteria for scientific soundness and importance, the risk-adjusted models performed well across groups of patients at varying levels of risk for admission and had excellent reliability in the random split sample. Moreover, the measures show that there was significant variation in performance within the Shared Savings Program ACO program, suggesting opportunities for improvement.
For more than a decade we have known that admission rates vary across the country, even after adjusting for differences in patient populations.27–30 To date, however, admission rates have been used as quality and accountability measures to only a limited degree.31 For example, it is only recently that CMS has started to use admission scores developed by the Agency for Healthcare Research and Quality, known as Prevention Quality Indicators, in several of its programs. However, these admission rates count only disease-specific admissions (e.g., heart failure admissions for heart failure patients) and do not capture the wide spectrum of hospital admissions (e.g., admissions related to falls or pneumonia) for which patients with chronic conditions are at increased risk.
As we move toward patient-centered systems of care and away from disease-centered systems of care, it is important to develop outcome measures that reflect the quality of comprehensive, coordinated care for patients with chronic conditions. The measures we developed use a broad outcome of acute, all-cause admissions to capture hospitalizations related to the two chronic conditions as well as those that are unrelated. Intentionally, the measures only include unplanned admissions in the outcome because most planned admissions do not represent acute events that could have been prevented by high-quality care.
The two measures of acute admissions are particularly suitable for helping ACOs advance their mission of improving quality of care and population health and of reducing cost growth. Diabetes and heart failure are complex, high-prevalence chronic conditions that increase the risk for hospital admission. Provision of coordinated care that is focused on improving health for the whole patient, across all stages of disease, and in the context of coexisting comorbidities and life circumstances should lower the risk of hospitalization. Indeed, research shows that effective health care can lower the risk of admission for these vulnerable groups of patients.3–8,32 In addition, providing timely and effective care that reduces the need for admissions is synergistic with the overall goal of ACOs to reduce costs, while maintaining high quality of care. While these acute admission measures may be applicable for comparing performance among other ambulatory systems of care, the measure specifications need to be vetted with relevant stakeholders.
In developing these outcome measures to assess ACO performance, it was important to identify factors influencing admission which are not markers of ACO quality, including differences in case-mix. This is challenging, as patients with chronic disease accumulate risk over decades. As such, we adjusted for clinical conditions that were present in the year prior to the start of the measurement period and we included indicators of diabetes and heart failure severity.
We did not adjust for sex in these outcome measures. After adjustment for age and clinical factors, any remaining differences in the risk for hospitalization among patients of different sex may represent disparities in care delivery and quality of care. Further, we tested adjustment for sex and found a very high degree of correlation in the measure scores adjusted and unadjusted for sex. This suggests that case-mix difference by sex contributes little to the variation in measure scores across ACOs. We did not adjust for SES given the ACO program design. The ACO program incentivizes providers to broadly address factors affecting health risks. Some of the pathways by which sociodemographic factors may influence admission rates can potentially be affected by ACOs. For example, ACOs can enhance access to care through transportation, expanded hours, and home visits. These measures, unadjusted for SES, will reveal when ACOs are doing exceptionally well with low-SES populations. Future research is needed to further examine the socioeconomic characteristics of ACO populations and their association with ACO performance.
Although we tailored our approach to the ACO program, our methods should inform future development of population-based measures designed to assess ambulatory care quality of other provider groups with shared responsibility for patient outcomes, such as providers in health plans, within states, and in other shared accountability arrangements.
Our measures have some limitations. First, these measures are based on claims data that may not fully capture disease severity and physical function. However, we have previously shown that claims-derived comorbidity data have good agreement with chart data.33–38 A related limitation is the concern that administrative data is vulnerable to coding intensity, wherein ACOs operating in higher utilization environments have more opportunities to document comorbid disease, increasing the severity of disease and the expected rates of admission. At the same time, health care utilization in the ambulatory care setting may also be a signal of quality; discerning these differences is an area for future research. Third, while the intention was to capture patients of all ages and with all stages of disease, we needed to exclude patients with LVADs, as these patients were small in prevalence and clustered among a few ACOs, making it difficult to risk adjust. Quality measures for this group of patients are needed. Fourth, there is no gold standard to assess validity. Fifth, the outcome of admission is subject to the effects of competing mortality; however, our denominator was based on days at risk for admission with time censored if patients’ died or were hospitalized.
In conclusion, we developed two measures of risk-standardized rates of acute hospital admissions among patients with diabetes and heart failure. These measures are consistent with consensus standards for publicly reported outcomes measures, and will incentivize ACOs to provide more efficient, coordinated, and partnered care for their patients in order to reduce the rates of acute, unplanned admissions. In these measures, we observed substantial variation in ACO performance, suggesting opportunities to improve care. Despite some limitations, the two measures of acute admission rates among patients with diabetes and heart failure are ready to be implemented to assess ACO performance.
Supplementary Material
Table 3.
| Final Risk-Adjustment Variables | Diabetes Final Model
|
Heart Failure Final Model
|
||||
|---|---|---|---|---|---|---|
| Rate Ratio | 95% Confidence Intervals | Rate Ratio | 95% Confidence Intervals | |||
| Age, per 1 y increase | 1.03 | 1.031 | 1.032 | |||
| Age group (65–70 y) | Reference | |||||
| Age group (70–80 y) | 1.04 | 1.030 | 1.046 | |||
| Age group (80–90 y) | 1.22 | 1.216 | 1.234 | |||
| Age group (> 90 y) | 1.42 | 1.402 | 1.428 | |||
| High-risk CV factors | 1.15 | 1.146 | 1.162 | 1.17 | 1.167 | 1.179 |
| Low-risk CV factors | 1.2 | 1.189 | 1.205 | 1.15 | 1.138 | 1.154 |
| Arrhythmia | 1.2 | 1.196 | 1.21 | 1.16 | 1.158 | 1.170 |
| Structural heart disease | 1.05 | 1.046 | 1.056 | |||
| Advanced cancer | 1.54 | 1.522 | 1.553 | 1.31 | 1.302 | 1.323 |
| Dementia | 1.25 | 1.245 | 1.263 | 1.09 | 1.083 | 1.095 |
| Heart failure | 1.52 | 1.508 | 1.527 | 1.18 | 1.172 | 1.182 |
| Diabetes with complications | 1.69 | 1.673 | 1.712 | |||
| Dialysis | 1.92 | 1.885 | 1.948 | 1.30 | 1.294 | 1.308 |
| Disability/frailty | 1.38 | 1.372 | 1.391 | 1.14 | 1.131 | 1.142 |
| Gastrointestinal and genitourinary disorders | 1.13 | 1.128 | 1.141 | 1.10 | 1.091 | 1.104 |
| Hematology | 1.12 | 1.112 | 1.131 | 1.11 | 1.105 | 1.125 |
| Infection and immune disorders | 1.2 | 1.190 | 1.22 | 1.28 | 1.276 | 1.289 |
| Kidney disease | 1.24 | 1.227 | 1.245 | 1.26 | 1.244 | 1.278 |
| Liver disease | 1.46 | 1.438 | 1.487 | 1.11 | 1.106 | 1.117 |
| Neurological disease | 1.09 | 1.078 | 1.092 | 1.21 | 1.200 | 1.211 |
| Psychiatric illness/substance abuse | 1.31 | 1.298 | 1.313 | 1.41 | 1.404 | 1.418 |
| Pulmonary disease | 1.38 | 1.369 | 1.384 | 1.37 | 1.360 | 1.375 |
| Other advanced organ failure | 1.44 | 1.431 | 1.456 | 1.09 | 1.082 | 1.094 |
| Diabetes severity index | 1.09 | 1.092 | 1.098 | |||
| CRT/ICD/pacemaker | 1.09 | 1.082 | 1.094 | |||
| Iron deficiency anemia | 1.17 | 1.165 | 1.179 | 1.16 | 1.159 | 1.170 |
| Major organ transplant | 1.31 | 1.248 | 1.368 | 1.13 | 1.086 | 1.168 |
| Other organ transplant | 1.06 | 1.03 | 1.093 | 1.02 | 0.992 | 1.039 |
| Hip fracture/major fracture | 1.05 | 1.04 | 1.066 | |||
CRT indicates cardiac resynchronization therapy; CV, cardiovascular; ICD, implantable cardiac defibrillator.
Acknowledgments
E.S.S. is supported by the Agency for Healthcare Research and Quality Patient Centered Outcomes Research (PCOR) Institutional Mentored Career Development Program (K12 HS023000). K.J.L. and J.S.R. are supported by the National Institute on Aging (K23 AG048359 and K08 AG032886) and by the American Federation for Aging Research through the Paul B Beeson Career Development Award Program. H.M.K. is supported by grant U01 HL105270-05 (Center for Cardiovascular Outcomes Research at Yale University) from the National Heart, Lung, and Blood Institute. Dr Krumholz is supported by grant U01 HL105270-05 (Center for Cardiovascular Outcomes Research at Yale University) from the National Heart, Lung, and Blood Institute.
All authors work under contract with the Centers for Medicare and Medicaid Services (CMS) to develop and maintain performance measures. The analyses on which this publication is based were performed under contract #HHSM-500-2012-00025I, Task Order HHSM-500-T0002, entitled, “Measure & Instrument Development and Support (MIDS)—Development and Reevaluation of the CMS Hospital Outcomes and Efficiency Measures,” funded by CMS, an agency of the US Department of Health and Human Services (HHS). The content of this publication does not necessarily reflect the views or policies of HHS. CMS reviewed and approved the use of its data for this work and approved submission of the manuscript. H.M.K. and J.S.R. disclose that they are the recipient of research agreements from Medtronic Inc. and Johnson & Johnson, through Yale University, to develop methods of clinical trial data sharing. J.S.R. also receives research support from the Food and Drug Administration (FDA) and Medtronic to develop methods for postmarket surveillance of medical devices. H.M.K. is chair of a cardiac scientific advisory board for UnitedHealth.
References
- 1.US Department of Health Human Services (HHS) Report to Congress: National strategy for quality improvement in health care. 2011. Published March. [Google Scholar]
- 2.Centers for Medicare and Medicaid Services (CMS) [Accessed March 21, 2014];Shared Savings Program. 2014 http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/sharedsavingsprogram/index.html.
- 3.Levine S, Steinman BA, Attaway K, Jung T, Enguidanos S. Home care program for patients at high risk of hospitalization. American Journal of Managed Care. 2012;18(8):e269–276. [PMC free article] [PubMed] [Google Scholar]
- 4.Zhang NJ, Wan TT, Rossiter LF, Murawski MM, Patel UB. Evaluation of chronic disease management on outcomes and cost of care for Medicaid beneficiaries. Health policy (Amsterdam, Netherlands) 2008;86(2–3):345–354. doi: 10.1016/j.healthpol.2007.11.011. [DOI] [PubMed] [Google Scholar]
- 5.Sommers LS, Marton KI, Barbaccia JC, Randolph J. Physician, nurse, and social worker collaboration in primary care for chronically ill seniors. Archives of internal medicine. 2000;160(12):1825–1833. doi: 10.1001/archinte.160.12.1825. [DOI] [PubMed] [Google Scholar]
- 6.Dorr DA, Wilcox AB, Brunker CP, Burdon RE, Donnelly SM. The effect of technology-supported, multidisease care management on the mortality and hospitalization of seniors. Journal of the American Geriatrics Society. 2008;56(12):2195–2202. doi: 10.1111/j.1532-5415.2008.02005.x. [DOI] [PubMed] [Google Scholar]
- 7.Chan CL, You HJ, Huang HT, Ting HW. Using an integrated COC index and multilevel measurements to verify the care outcome of patients with multiple chronic conditions. BMC health services research. 2012;12:405. doi: 10.1186/1472-6963-12-405. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Littleford A, Kralik D. Making a difference through integrated community care for older people. Journal of Nursing and Healthcare of Chronic Illness. 2010;2(3):178–186. [Google Scholar]
- 9.Brown RS, Peikes D, Peterson G, Schore J, Razafindrakoto CM. Six features of Medicare coordinated care demonstration programs that cut hospital admissions of high-risk patients. Health affairs (Project Hope) 2012;31(6):1156–1166. doi: 10.1377/hlthaff.2012.0393. [DOI] [PubMed] [Google Scholar]
- 10.Centers for Medicare & Medicaid Services (CMS) [Accessed March 27, 2014];Medicare Health Support. 2012 https://www.cms.gov/Medicare/Medicare-General-Information/CCIP/
- 11.Centers for Medicare & Medicaid Services (CMS), HHS. Medicare program; revisions to payment policies under the Physician Fee Schedule, Clinical Laboratory Fee Schedule, access to identifiable data for the Center for Medicare and Medicaid Innovation Models & other revisions to Part B for CY 2015. Final rule with comment period. 792014:67547. [PubMed] [Google Scholar]
- 12.(NQF) NQF. [Accessed February 16, 2015];Review and Update of Guidance for Evaluating Evidence and Measure Testing - Technical Report. 2013 http://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=74076.
- 13.Krumholz HM, Brindis RG, Brush JE, et al. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council. Endorsed by the American College of Cardiology Foundation. Circulation. 2006;113(3):456–462. doi: 10.1161/CIRCULATIONAHA.105.170769. [DOI] [PubMed] [Google Scholar]
- 14.Horwitz LI, Grady JN, Cohen DB, et al. Development and Validation of an Algorithm to Identify Planned Readmissions From Claims Data. Journal of hospital medicine: an official publication of the Society of Hospital Medicine. 2015;10(10):670–677. doi: 10.1002/jhm.2416. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Pope GCKJ, Ingber MJ, Freeman S, Sekar R, Newhart C. Evaluation of the CMS-HCC Risk Adjustment Model. 2011 http://www.cms.gov/Medicare/Health-Plans/MedicareAdvtgSpecRateStats/downloads/evaluation_risk_adj_model_2011.pdf.
- 16.Chang HY, Weiner JP, Richards TM, Bleich SN, Segal JB. Validating the adapted Diabetes Complications Severity Index in claims data. The American journal of managed care. 2012;18(11):721–726. [PubMed] [Google Scholar]
- 17.Young BA, Lin E, Von Korff M, et al. Diabetes complications severity index and risk of mortality, hospitalization, and healthcare utilization. The American journal of managed care. 2008;14(1):15–23. [PMC free article] [PubMed] [Google Scholar]
- 18.Burstein JM, Yan R, Weller I, Abramson BL. Management of congestive heart failure: a gender gap may still exist. Observations from a contemporary cohort. BMC cardiovascular disorders. 2003;3:1. doi: 10.1186/1471-2261-3-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Rossi MC, Cristofaro MR, Gentile S, et al. Sex disparities in the quality of diabetes care: biological and cultural factors may play a different role for different outcomes: a cross-sectional observational study from the AMD Annals initiative. Diabetes care. 2013;36(10):3162–3168. doi: 10.2337/dc13-0184. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Wexler DJ, Grant RW, Meigs JB, Nathan DM, Cagliero E. Sex disparities in treatment of cardiac risk factors in patients with type 2 diabetes. Diabetes care. 2005;28(3):514–520. doi: 10.2337/diacare.28.3.514. [DOI] [PubMed] [Google Scholar]
- 21.Krumholz HM, Bernheim SM. Considering the role of socioeconomic status in hospital outcomes measures. Annals of internal medicine. 2014;161(11):833–834. doi: 10.7326/M14-2308. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.National Quality Forum (NQF) Risk Adjustment for Socioeconomic Status or Other Sociodemographic Factors. [Accessed February 16, 2015];Technical Report. 2014 http://www.qualityforum.org/Publications/2014/08/Risk_Adjustment_for_Socioeconomic_Status_or_Other_Sociodemographic_Factors.aspx.
- 23.Mazerolle MJ. [Accessed 3/14/2014, 2014];Making sense out of Akaike’s Information Criterion (AIC): its use and interpretation in model selection and inference from ecological data. [Google Scholar]
- 24.Cameron ACWF. R-Squared Measures for Count Data Regression Models with Applications to Health-Care Utilization. Journal of Business & Economic Statistic. 1996;14(2):209–220. [Google Scholar]
- 25.Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin. 1979;86(2):420–428. doi: 10.1037//0033-2909.86.2.420. [DOI] [PubMed] [Google Scholar]
- 26.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. [PubMed] [Google Scholar]
- 27.Wennberg JE, Freeman JL, Culp WJ. Are hospital services rationed in New Haven or over-utilised in Boston? Lancet. 1987;1(8543):1185–1189. doi: 10.1016/s0140-6736(87)92152-0. [DOI] [PubMed] [Google Scholar]
- 28.Fisher ES, Wennberg JE, Stukel TA, Sharp SM. Hospital readmission rates for cohorts of Medicare beneficiaries in Boston and New Haven. The New England journal of medicine. 1994;331(15):989–995. doi: 10.1056/NEJM199410133311506. [DOI] [PubMed] [Google Scholar]
- 29.Wennberg JE, Freeman JL, Shelton RM, Bubolz TA. Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. The New England journal of medicine. 1989;321(17):1168–1173. doi: 10.1056/NEJM198910263211706. [DOI] [PubMed] [Google Scholar]
- 30.Wennberg JECM, editor. The quality of medical care in the United States: a report on the Medicare program. American Hospital Association Press; 1999. [PubMed] [Google Scholar]
- 31.RTI International, Telligen. Accountable Care Organization 2013 Program Analysis: Quality Performance Standards Narrative Measure Specifications. 2012. [Google Scholar]
- 32.Tehrani DM, Russell D, Brown J, et al. Discord among performance measures for central line-associated bloodstream infection. Infection control and hospital epidemiology: the official journal of the Society of Hospital Epidemiologists of America. 2013;34(2):176–183. doi: 10.1086/669090. [DOI] [PubMed] [Google Scholar]
- 33.Krumholz HM, Lin Z, Drye EE, et al. An administrative claims measure suitable for profiling hospital performance based on 30-day all-cause readmission rates among patients with acute myocardial infarction. Circulation. Cardiovascular quality and outcomes. 2011;4(2):243–252. doi: 10.1161/CIRCOUTCOMES.110.957498. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Lindenauer PK, Normand SL, Drye EE, et al. Development, validation, and results of a measure of 30-day readmission following hospitalization for pneumonia. Journal of hospital medicine: an official publication of the Society of Hospital Medicine. 2011;6(3):142–150. doi: 10.1002/jhm.890. [DOI] [PubMed] [Google Scholar]
- 35.Keenan PS, Normand SL, Lin Z, et al. An administrative claims measure suitable for profiling hospital performance on the basis of 30-day all-cause readmission rates among patients with heart failure. Circulation. Cardiovascular quality and outcomes. 2008;1(1):29–37. doi: 10.1161/CIRCOUTCOMES.108.802686. [DOI] [PubMed] [Google Scholar]
- 36.Bratzler DW, Normand SL, Wang Y, et al. An administrative claims model for profiling hospital 30-day mortality rates for pneumonia patients. PloS one. 2011;6(4):e17401. doi: 10.1371/journal.pone.0017401. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Krumholz HM, Wang Y, Mattera JA, et al. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction. Circulation. 2006;113(13):1683–1692. doi: 10.1161/CIRCULATIONAHA.105.611186. [DOI] [PubMed] [Google Scholar]
- 38.Krumholz HM, Wang Y, Mattera JA, et al. An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation. 2006;113(13):1693–1701. doi: 10.1161/CIRCULATIONAHA.105.611194. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.