Performance Differences in Year 1 of Pioneer Accountable Care Organizations

Abstract

BACKGROUND

In 2012, a total of 32 organizations entered the Pioneer accountable care organization (ACO) program, in which providers can share savings with Medicare if spending falls below a financial benchmark. Performance differences associated with characteristics of Pioneer ACOs have not been well described.

METHODS

In a difference-in-differences analysis of Medicare fee-for-service claims, we compared Medicare spending for beneficiaries attributed to Pioneer ACOs (ACO group) with other beneficiaries (control group) before (2009 through 2011) and after (2012) the start of Pioneer ACO contracts, with adjustment for geographic area and beneficiaries’ sociodemographic and clinical characteristics. We estimated differential changes in spending for several subgroups of ACOs: those with and those without clear financial integration between hospitals and physician groups, those with higher and those with lower baseline spending, and the 13 ACOs that withdrew from the Pioneer program after 2012 and the 19 that did not.

RESULTS

Adjusted Medicare spending and spending trends were similar in the ACO group and the control group during the precontract period. In 2012, the total adjusted per-beneficiary spending differentially changed in the ACO group as compared with the control group (−$29.2 per quarter, P = 0.007), consistent with a 1.2% savings. Savings were significantly greater for ACOs with baseline spending above the local average, as compared with those with baseline spending below the local average (P = 0.05 for interaction), and for those serving high-spending areas, as compared with those serving low-spending areas (P = 0.04). Savings were similar in ACOs with financial integration between hospitals and physician groups and those without, as well as in ACOs that withdrew from the program and those that did not.

CONCLUSIONS

Year 1 of the Pioneer ACO program was associated with modest reductions in Medicare spending. Savings were greater for ACOs with higher baseline spending than for those with lower baseline spending and were unrelated to withdrawal from the program. (Funded by the National Institute on Aging and others.)

In a substantial departure from fee-for-service payment, 32 provider organizations with diverse organizational forms and service areas entered the Medicare Pioneer accountable care organization (ACO) program in 2012. Under the Pioneer model, ACOs share in savings with Medicare if spending for an attributed patient population falls sufficiently below a financial benchmark and incur losses if spending sufficiently exceeds the benchmark.¹ The benchmark is based on spending for the attributed population at baseline, trended forward by national spending growth. The proportion of savings gained (or losses borne) by an ACO depends on its performance on 33 quality measures.^2–4 In the first year, ACOs were required only to report on these measures to be eligible for maximum savings.

Although initial reports suggest that early savings were achieved by Pioneer ACOs,^5,6 it is still unclear whether performance differs systematically according to organizational or market-level factors. For certain factors, the presence or absence of performance differences has important policy implications.^7,8 In particular, the financial integration of outpatient practices with hospitals could enhance care coordination and savings or, conversely, could limit savings owing to weaker incentives to constrain inpatient care, with different implications for the extent of provider consolidation encouraged by ACO programs.^9–11

In addition, prior spending in an ACO could be an important predictor of its future savings, because a reduction in spending may be more difficult for efficient organizations, which have fewer opportunities to cut wasteful care, than it would be for organizations that are less efficient initially.¹² Among the 13 organizations that left the Pioneer program in 2013 or 2014, commonly reported reasons for withdrawal included unsustainability of the financial model for already-efficient organizations and concerns that savings that are determined by the Centers for Medicare and Medicaid Services (CMS) in comparisons with benchmarks may underestimate the actual savings achieved by ACOs.^13–16 For example, ACOs facing high local spending growth may be disadvantaged by the use of national growth rates to set benchmarks.¹⁷ Understanding how baseline spending and program withdrawal relate to actual savings could inform regulatory efforts to enhance incentives for organizations to participate and to limit spending.^18,19

In analyses of Medicare claims data from 2009 through 2012, we compared per-beneficiary Medicare spending and performance on several quality measures between Pioneer ACOs and local nonparticipating providers before versus after the start of Pioneer ACO contracts in 2012. We compared savings estimated by this approach between policy-relevant subgroups of ACOs.

METHODS

STUDY POPULATION

We analyzed Medicare claims from 2009 through 2012 for a random 20% sample of beneficiaries; in a given year, this sample included sample members from the prior year plus a 20% sample of newly eligible beneficiaries. In each study year, we limited the sample to beneficiaries who were continuously enrolled in fee-for-service Medicare (or, in the case of beneficiaries who died during the study year, those who were continuously enrolled during the time they were alive). For each of the 32 Pioneer ACOs, we used publicly accessible databases to match the names of participating physicians, practices, and facilities posted by the ACO to National Provider Identifiers (NPIs) or taxpayer identification numbers (TINs) (see the Supplementary Appendix, available with the full text of this article at NEJM.org).^20–23

We attributed each beneficiary in each year to the ACO or non-ACO TIN that accounted for the most allowed charges for primary care services received by the beneficiary during the year (see the Supplementary Appendix).^23,24 Because Pioneer ACOs may select individual physicians from practices for inclusion in contracts, we defined ACOs as collections of physician NPIs for our main analysis. Because this approach held ACO physicians constant over the study period, we conducted a sensitivity analysis allowing physician turnover within ACO practices by defining ACOs as collections of TINs; TINs generally identify the practice or larger provider organizations for physicians practicing in groups (see the Supplementary Appendix).²³

We classified beneficiaries attributed to organizations that entered the Pioneer program in 2012 as the ACO group and beneficiaries attributed to non-ACO TINs who were living in hospital referral regions (HRRs) served by Pioneer ACOs as the control group. We excluded beneficiaries attributed to organizations entering the Medicare Shared Savings Program (MSSP) in 2012, which were defined similarly as collections of NPIs or TINs with the use of lists of constituent providers posted by CMS,²⁵ because these groups also faced new incentives to limit spending in 2012. We did not expect MSSP ACOs to achieve meaningful savings in 2012 because they were exposed to weaker incentives, as compared with those in the Pioneer program, and for only part of the year, but their early efforts could have affected spending in the control group slightly. Finally, we excluded an average of 14.2% of the beneficiaries in each study year because they received no primary care services in the year to support attribution (see the Supplementary Appendix).

STUDY VARIABLES

Spending and Quality Measures

For each beneficiary in each quarter, we summed Medicare payments for all services reimbursed under Part A or Part B, following Medicare ACO program specifications,²⁶ and categorized spending according to the care setting and type of service. ACO contracts include 33 quality measures related to patients’ experiences, care coordination, preventive care, and disease control. Building on our previous analysis of patients’ experiences in ACOs,²³ we examined 4 additional contract measures related to care coordination and preventive care that could be assessed from claims: screening mammography for women 65 to 69 years of age, all-cause 30-day readmissions, and hospitalizations for two ambulatory care–sensitive conditions (ACSCs; conditions for which appropriate ambulatory care could potentially reduce the need for inpatient care) — chronic obstructive pulmonary disease and congestive heart failure.^4,27 We could not analyze changes in other contract measures because they were not captured by claims and ACOs did not report uniformly on these measures until 2012. Because cardiovascular disease and diabetes are the focus of 12 of these measures, we also assessed hospitalizations for ACSCs related to these conditions^27,28 and several preventive services for beneficiaries with diabetes: glycated hemoglobin testing, low-density lipoprotein cholesterol testing, and diabetic retinal examinations. We specified hospitalizations and readmissions as annual counts on the basis of admission dates and assessed the receipt of preventive services on an annual basis (even those with 2-year screening intervals) to align with the single postcontract year analyzed in our study.

ACO Characteristics

We selected characteristics that have direct policy implications, did not require adjustment for other ACO characteristics to support conclusions, and were shared by a substantial proportion of Pioneer ACOs. On the basis of CMS descriptions of Pioneer ACOs,²⁹ we distinguished organizations that were likely to negotiate commercial prices on behalf of financially integrated hospitals and physician practices (integrated delivery systems and physician–hospital organizations) from organizations without vertically integrated contracting arrangements clearly in place (independent medical groups or partnerships between financially independent medical groups and hospitals). In a sensitivity analysis, we categorized ACOs on the basis of a claims-based measure of hospital ownership of physician practices (see the Supplementary Appendix).

We constructed two measures related to baseline spending. First, we categorized each ACO according to whether the mean risk-adjusted Medicare spending for the control group in its service area was above or below the median among Pioneer ACO service areas (see the Supplementary Appendix). Second, we categorized each ACO according to whether the mean risk-adjusted spending for its attributed beneficiaries was above or below the mean risk-adjusted spending for the control group in its service area (see the Supplementary Appendix).

For the two baseline spending measures, we used data from 2008 (before the study period) to mitigate bias from regression to the mean. For both measures, the ACOs in the higher spending categories in 2008 continued to show higher spending in the period from 2009 through 2011 than did the ACOs in the lower spending categories in 2008, and we found no evidence of regression to the mean in these baseline spending differences over the period from 2009 through 2011 (see the Supplementary Appendix). Finally, we identified ACOs that withdrew from the Pioneer program after 2012.

Covariates

From Medicare Beneficiary Summary Files, we collected information on age, sex, and race or ethnic group^30,31 and assessed whether disability was the original reason for eligibility and whether beneficiaries had end-stage renal disease. From the Chronic Conditions Data Warehouse (CCW), which draws from diagnoses since 1999 to describe beneficiaries’ accumulated disease burden,³² we assessed whether the beneficiaries had any of 25 conditions in the CCW by the start of each study year. From claims in the preceding year, we also calculated Hierarchical Condition Category risk scores, which predicted Medicare spending for each beneficiary in each study year.³³ Finally, we assessed area-level sociodemographic characteristics from U.S. Census data,³⁴ and, using a validated claims-based algorithm, we assessed whether beneficiaries were longterm nursing home residents.³⁵

STATISTICAL ANALYSIS

We used linear regression and a difference-indifferences approach to estimate changes in spending or quality in the ACO group from the precontract period to the postcontract period that differed from concurrent changes in the control group and were not explained by geographic area or changes in observed sociodemographic and clinical characteristics of beneficiaries. Specifically, for each dependent variable, we fitted the following model:

• E(Y_i,t,k,h) = β₀ + β₁ACO_indicators_k+β₂Time_indicators_t+β₃(ACO_Group_k × 2012_t)+β₄HRR_indicators_h+β₅(HRR_indicators_h × Time_indicators_t)+β₆Covariates_i,t,

where E denotes the expected value, Y_i,t,k,h is the spending (or quality measure) for beneficiary i at time t assigned to ACO or non-ACO TIN k and residing in HRR h, ACO_indicators is a vector of ACO indicators, Time_indicators is a vector of quarter indicators (or year indicators in models of quality, omitting a reference time), ACO_Group indicates that the beneficiary was attributed to an organization entering the Pioneer ACO program in 2012, the term 2012 indicates the year 2012, HRR_indicators is a vector of HRR indicators (omitting a reference region), and Covariates denotes the beneficiary characteristics in Table 1 (with age specified as a categorical variable and CCW conditions as 25 indicators). We included HRR and time indicators, and their interaction, to compare each beneficiary attributed to an ACO with beneficiaries in the control group living in the same area and to adjust for area-specific trends in spending or quality that were evident in the control group.

Table 1.

Characteristics of Beneficiaries in Pioneer Accountable Care Organizations (ACOs) and the Control Group, before and after the Start of ACO Contracts in 2012.^*

Characteristic	Precontract Period, 2009–2011 (N = 14,876,933 beneficiary-yr)		Postcontract Period, 2012 (N = 5,043,581 beneficiary-yr)		Differential Change for ACO Group vs. Control Group
	ACO Group (N = 566,410)	Control Group (N = 14,310,523)	ACO Group (N = 201,644)	Control Group (N = 4,841,937)
Age (yr)	71.1±12.4	71.4±12.2	71.0±12.3	71.2±12.3	0.1
Female sex (%)	57.9	57.2	57.6	56.9	0.0
Race or ethnic group (%)†
White	81.7	82.6	81.1	82.1	0.0
Black	9.3	8.9	9.4	9.1	−0.1
Hispanic	6.0	5.0	6.0	5.2	−0.2
Other	3.0	3.4	3.5	3.7	0.2‡
Medicaid recipient (%)	16.5	16.6	16.1	16.2	0.0
Disabled (%)§	22.6	22.5	23.1	23.3	−0.3
End-stage renal disease (%)	1.2	1.3	1.2	1.3	0.0
Long-term nursing home resident (%)	2.5	3.1	2.3	3.0	0.0
CCW conditions¶
No. of conditions	5.2±3.4	5.3±3.3	5.4±3.5	5.5±3.5	0.0
≥6 Conditions (%)	43.7	44.8	45.3	46.5	−0.2
≥9 Conditions (%)	17.3	18.0	18.9	19.8	−0.2
HCC risk score‖	1.2±1.1	1.2±1.1	1.2±1.1	1.3±1.1	0.0
ZCTA-level characteristic
% Below federal poverty level	8.9	9.2	8.8	9.1	0.0
% With high-school diploma	76.3	75.5	76.4	75.6	0.1
% With college degree	20.6	19.8	20.8	19.9	0.1

^*Plus–minus values are means ±SD. The control group was defined as beneficiaries attributed to non-ACO taxpayer identification numbers who were living in hospital referral regions served by Pioneer ACOs. There were no significant between-group differences in the precontract period, except for differences in sex (P = 0.02) and long-term nursing home residence (P = 0.003). Means and percentages were adjusted for geographic area to reflect comparisons within hospital referral regions. ZCTA denotes ZIP Code tabulation area.

^†Race or ethnic group was determined from Medicare enrollment files.

^‡Result was significantly different from zero (P = 0.02).

^§Data indicate the percentage of respondents for whom disability was the original reason for Medicare eligibility.

^¶Chronic conditions from the Chronic Conditions Data Warehouse (CCW) include the following 25 conditions: acute myocardial infarction, Alzheimer’s disease, Alzheimer’s disease and related disorders or senile dementia, anemia, asthma, atrial fibrillation, benign prostatic hyperplasia, chronic kidney disease, chronic obstructive pulmonary disease, depression, diabetes, heart failure, hip or pelvic fracture, hyperlipidemia, hypertension, hypothyroidism, ischemic heart disease, osteoporosis, rheumatoid arthritis or osteoarthritis, stroke or transient ischemic attack, breast cancer, colorectal cancer, endometrial cancer, lung cancer, and prostate cancer.

^‖Hierarchical Condition Categories (HCC) risk scores are derived from demographic and diagnostic data in Medicare enrollment and claims files, with higher scores indicating higher predicted spending in the subsequent year. For each beneficiary in each study year, we assessed the HCC risk score on the basis of enrollment and claims data in the prior year. In our study, HCC risk scores ranged from 0.12 to 17.16, with 90% of beneficiaries having a score of 2.55 or less.

Thus, β₃ is the adjusted mean differential change in spending for beneficiaries attributed to ACOs relative to local changes in spending for the control group (estimated savings or losses). In contrast to comparisons with benchmarks used by CMS to calculate savings, our analysis was intended to estimate savings or losses by comparing spending in 2012 for ACO-attributed patients with spending that would be expected in the absence of ACO contracts, with the use of local changes in a similar population to establish that counterfactual scenario. In supplementary analyses, we examined factors potentially contributing to differences between savings estimated by our approach and savings determined from comparisons with benchmarks set by CMS (see the Supplementary Appendix).

In prespecified subgroup analyses, we compared differential changes between ACO subgroups by adding interactions between the β₃ term and each ACO characteristic. We modeled these interactions jointly for all the ACO characteristics except for withdrawal from the Pioneer program, which we modeled separately to obtain estimates unadjusted for other ACO characteristics.

We conducted several analyses to address potential sources of bias. First, we tested and adjusted for any differences in spending trends between the ACO group and the control group during the precontract period. Specifically, we estimated differential changes in spending under the assumption that trend differences would have continued through 2012 if there were no differential change in the ACO group (see the Supplementary Appendix).

Second, we conducted falsification tests alternately treating 2010 and 2011 as the postcontract year to determine whether factors causing year-to-year changes in Medicare spending in the precontract period (e.g., fee changes) were associated with differential spending changes in the ACO group.³⁶ Third, we compared sociodemographic and clinical characteristics between the ACO group and the control group before versus after the start of Pioneer ACO contracts in 2012 to gauge potential bias from differential changes in related characteristics that we could not measure. In all the analyses, we used robust variance estimators to account for clustering within ACOs (for the ACO group) or HRRs (for the control group).³⁷ Finally, using logistic regression, we analyzed annual indicators of one or more readmissions or one or more hospitalizations for an ACSC.

RESULTS

STUDY POPULATION

Our study sample included 14,876,933 beneficiary-years from 2009 through 2011 and 5,043,581 beneficiary-years in 2012; of these, 566,410 and 201,644 beneficiary-years, respectively, were in the ACO group. Within geographic areas, observed characteristics of the beneficiaries in the ACO group and those in the control group were similar in the precontract period, as were changes in these characteristics in 2012 (Table 1).

CHANGES IN SPENDING AND QUALITY

In the precontract period, the adjusted quarterly total Medicare spending per beneficiary was similar in the ACO group and the control group ($18.7 per quarter lower in the ACO group, P = 0.37) and grew at similar rates ($0.7 per quarter slower in the ACO group, P = 0.68). In 2012, the mean total per-beneficiary spending in the ACO group differentially changed as compared with the control group (unadjusted differential change, −$29.8 per quarter; P = 0.03; adjusted differential change, −$29.2 per quarter; P = 0.007), constituting a 1.2% reduction relative to an expected quarterly mean of $2,455.8 in 2012 for the ACO group (Table 2, and Table S1 in the Supplementary Appendix).

Table 2.

Differential Changes in Spending and Quality for Pioneer ACOs versus the Control Group.^*

Measure	Mean in the ACO Group†	Difference in ACO Group vs. Control Group in Precontract Period	P Value‡	Differential Change in Postcontract Period for ACO vs. Control Group	P Value‡
Total quarterly per-beneficiary spending ($)	2,455.8	−18.7	0.37	−29.2	0.007
Quarterly per-beneficiary spending according to type of service and care setting ($)§
Total acute inpatient care	911.2	3.9	0.71	−13.5	0.04
Facility¶	792.9	3.4	0.71	−12.2	0.04
Professional services	118.3	0.5	0.75	−1.3	0.13
Total outpatient care	793.4	−14.1	0.17	−6.9	0.24
Office	405.0	−24.6	0.01	7.3	0.02
Hospital outpatient department	388.4	10.5	0.44	−14.2	<0.001
Total post-acute care	270.6	−0.4	0.93	−8.7	0.003
Facility	256.7	−0.4	0.91	−8.5	0.003
Skilled nursing facility	204.7	−1.2	0.77	−6.5	0.01
Rehabilitation facility	52.0	0.8	0.52	−2.0	0.23
Professional services	13.9	0.0	0.93	−0.2	0.47
Home health care	150.8	5.5	0.18	0.9	0.66
Durable medical equipment	80.2	0.7	0.74	−1.6	0.10
Hospice	50.9	−1.8	0.24	0.0	0.97
Annual quality measure‖
30-day readmissions (no.)	0.26	−0.02	0.73	0.00	0.78
Hospitalizations for ACSCs (no.)	0.06	0.00	0.85	0.00	0.73
Congestive heart failure	0.02	0.00	0.16	0.00	0.08
COPD or asthma	0.01	0.00	0.07	0.00	0.18
Cardiovascular disease or diabetes	0.02	0.00	0.45	0.00	0.81
Screening mammography for women 65–69 yr of age (%)	55.2	1.6	0.008	0.0	0.90
Preventive services for beneficiaries with diabetes (%)
Glycated hemoglobin testing	73.1	1.8	<0.001	0.5	0.006
Low-density lipoprotein cholesterol testing	77.4	1.6	0.04	0.5	0.05
Diabetic retinal examination	55.2	0.2	0.71	0.8	0.005
Received all three services	38.5	1.4	0.03	0.8	0.009

^*COPD denotes chronic obstructive pulmonary disease.

^†Values were calculated by adding the adjusted precontract difference between the ACO group and the control group to the unadjusted quarterly mean in 2012 for the control group to approximate the expected quarterly mean for the ACO group in 2012 in the absence of a differential change. Thus, dividing the differential change by this expected quarterly mean yields an estimate in relative terms.

^‡P values are for the comparison with zero.

^§We report the results for main categories of spending. Estimates for each category do not sum to total spending because we did not analyze lesser contributions separately from other categories of services or miscellaneous care settings (e.g., ambulatory surgical centers). Professional services are physician and ancillary services appearing in the carrier claims file and reimbursed under Part B. Spending by an acute or postacute care facility was defined as the portion of acute or postacute care spending that was reimbursed under Part A. Outpatient care in office settings included safety-net settings. Analysis of readmissions was limited to hospitalized beneficiaries.

^¶Inpatient-facility spending did not include capital payments, disproportionate share hospital payments, or indirect medical-education payments. To further adjust for between-hospital differences in Medicare payments for admissions in the same diagnosis-related group, we standardized inpatient-facility spending by calculating a national mean payment for each diagnosis-related group and summing mean payments across admissions for each beneficiary rather than using actual Medicare payments. Estimates were not appreciably changed by this standardization, and we report the standardized estimates in the table.

^‖Hospitalizations for ambulatory care–sensitive conditions (ACSCs; conditions for which appropriate ambulatory care could potentially reduce the need for inpatient care) that were related to cardiovascular disease or diabetes included Agency for Healthcare Research and Quality (AHRQ) Prevention Quality Indicators (PQIs) of hospitalization for uncontrolled diabetes, short-term complications of diabetes, long-term complications of diabetes, lower-extremity amputation, hypertension, angina without procedure, and congestive heart failure.^27,28 Estimates for all the hospitalizations for ACSCs also included hospitalization for COPD or asthma and other conditions assessed by AHRQ PQIs that are relevant to the Medicare population (dehydration, bacterial pneumonia, and urinary tract infection). Preventive services for diabetes were assessed among beneficiaries with a diagnosis of diabetes mellitus before the study year according to the CCW.

This differential change in total spending included estimated savings in spending on acute inpatient care, hospital outpatient care, and post-acute care, particularly in skilled nursing facilities (Table 2). In contrast, spending on outpatient care in office settings differentially increased in 2012 for the ACO group, partially offsetting the lower spending on hospital outpatient care. As compared with the control group, changes in performance on quality measures in the ACO group suggested either significant but small improvements or no significant differential change (Table 2).

PERFORMANCE DIFFERENCES

Differential changes in total spending were similar in ACOs with financial integration between hospitals and physician groups and in those without such financial integration, as defined by ACO descriptions (Fig. 1)²⁹ or a claims-based measure of hospital ownership of physician practices (see the Supplementary Appendix). Estimated savings were greater for ACOs with baseline spending above the local average than for those with baseline spending below the local average ($39.4 per quarter more in savings, P = 0.05 for interaction) and greater for ACOs serving high-spending areas than for those serving low-spending areas ($56.3 per quarter more in savings, P = 0.04 for interaction). Estimated savings in the 13 ACOs that withdrew from the Pioneer program after 2012 (−$33.0 per quarter, P = 0.04) were similar to the savings in the 19 ACOs that remained in the program (−$26.1 per quarter, P = 0.08) (P = 0.75 for interaction).

Figure 1. Differential Changes in Medicare Spending for Pioneer Accountable Care Organizations (ACOs) versus the Control Group, According to ACO Characteristics.

Adjusted differential changes in total quarterly Medicare spending from the precontract period (2009 through 2011) to the postcontract period (2012) for beneficiaries attributed to Pioneer ACOs versus the control group (beneficiaries attributed to non-ACO taxpayer identification numbers who were living in hospital referral regions served by Pioneer ACOs) are shown for subgroups of ACOs defined according to financial integration between hospitals and physician groups versus no financial integration between those entities, baseline (2008) spending for the control group in ACO service areas, ACO baseline spending relative to local average spending in the control group, and withdrawal from or continued participation in the Pioneer program. Estimates are shown with 95% confidence intervals (error bars). P values for tests of differences in savings between ACO subgroups (interaction tests) are shown. Because of correlations between estimates, differences between subgroups may be significant even if confidence intervals overlap.

SENSITIVITY ANALYSES

Adjustment for spending trends in the precontract period did not affect our main conclusions (see the Supplementary Appendix for discussion of trend-adjusted estimates). Results were substantively similar when ACOs were defined as collections of TINs instead of NPIs. In falsification tests, spending changes in 2010 and in 2011 did not differ significantly between the ACO group and the control group. The use of logistic regression for analyses of quality measures did not alter the conclusions.

DISCUSSION

As compared with changes for other beneficiaries in ACO service areas, changes in Medicare spending for beneficiaries served by Pioneer ACOs were consistent with modest savings in the first year of the Pioneer program. When aggregated to the entire population attributed to Pioneer ACOs, our quarterly per-beneficiary estimate of −$29.2 suggests that total Medicare spending was approximately $118 million lower than expected, a sum that falls between the actuarial calculation of $87 million by the CMS and an estimate of $147 million in a previous evaluation.^5,38 Our estimate exceeds the $76 million in bonuses paid by CMS to Pioneer ACOs by $42 million.⁶

Contributing to these estimated savings were reductions in spending on acute and postacute care as well as an apparent substitution of care in lower-priced office settings for care in higher-priced hospital outpatient departments, as observed in one commercial ACO initiative.^39,40 The first year of the Pioneer program was not associated with significant changes in the rates of readmissions, hospitalizations for ACSCs, or screening mammography but was associated with slight increases in the rates of use of preventive services in diabetes care. Together with a recent study examining the experiences of patients in ACOs,²³ our findings suggest that Pioneer ACO contracts have been associated with some early savings and either unchanged or improved performance on quality measures.

Estimated savings were similar in ACOs with financial integration between hospitals and physician groups and in those without such financial integration, suggesting that this form of provider consolidation — which could increase bargaining power and prices in commercial markets⁴¹ — may not be necessary in order for ACOs to achieve lower spending in Medicare. Our findings also suggest that ACOs with higher baseline spending, whether it was due to less-efficient practices or unobserved differences in case mix, were able to reduce spending more easily, at least initially, than ACOs with lower baseline spending. In addition, we found that savings occurred among the 13 ACOs that subsequently withdrew from the Pioneer program, whereas CMS had calculated minimal savings or losses for 12 of these ACOs in comparisons with benchmarks.⁴² Supplementary analyses suggest that differences in local spending growth (faster for ACOs that withdrew than for those that remained in the program) as well as regression-to-the-mean effects introduced by prospective attribution in the Pioneer program probably contributed to differences between our estimates and the CMS estimates (see the Supplementary Appendix).^17,43

Taken together, these findings have important implications for payment policy in Medicare ACO programs. First, given the lack of a relationship between estimated savings and continued participation in the Pioneer program, sustaining or expanding participation in a Pioneer-like ACO program will probably require greater and more reliable rewards for ACOs that reduce spending than those currently in place. For example, our findings support the consideration of increased shared-savings rates and a benchmarking approach that would account for local spending growth and would sever or weaken the link between ACO benchmarks and savings in preceding contract periods; currently, this link diminishes incentives to achieve and maintain increased efficiency.^{12,17,19,44,45} Second, as analysts score proposed changes to ACO payment rules, they should consider lost savings from organizations that withdraw from the ACO programs in response to current incentives. Stronger incentives to participate in ACO programs would diminish the share of savings appropriated by Medicare for a given ACO but could lead to more ACOs generating savings.

Third, the continued use of historical spending for patients served by an ACO as the initial basis for its benchmark may be important to attract organizations with high initial spending, because their participation may be particularly valuable. Our findings also suggest, however, that benchmarks for ACOs with high spending could be gradually reduced, as compared with the benchmarks for ACOs with lower spending, without necessarily putting the high-spending ACOs at a comparative disadvantage. Constraining growth in benchmarks for ACOs with high spending could be important for establishing equitable benchmarks and fostering healthy competition among ACOs.^12,44

Our study had several limitations. First, although the characteristics of the patients differed minimally between the ACO group and the control group, the Pioneer program is voluntary, and program participants differ from nonparticipating providers in many respects. Baseline differences in spending between ACOs and non-ACO providers were minimal, however, and we adjusted for those differences by means of difference-in-differences comparisons. Second, organizations may have decided to participate in the Pioneer program because of ongoing or planned efforts to constrain spending. Similar spending trends between ACOs and other providers suggested that no such efforts were under way during the precontract period, however, and constraining fee-for-service spending would not have served the financial interests of the organizations without participation in alternative payment models. Thus, although our findings regarding Pioneer ACOs may not be generalizable to other provider organizations, they do suggest that differential spending changes were related to the start of ACO contracts.

Finally, our estimates of savings do not account for the costs to CMS of administering an ACO program or the costs to ACOs of implementing strategies to limit spending. Increased understanding of these costs, the evolution of ACO performance over time, and the extent to which Medicare ACO contracts affect care for patients served by ACOs but not covered by the contracts⁴⁶ will be needed to characterize the potential for long-term savings to society from Medicare ACO initiatives.