Determinants of Success in Shared Savings Programs: An Analysis of ACO and Market Characteristics

Abstract

Background

Medicare’s Accountable Care Organization (ACO) programs introduced shared savings to traditional Medicare, which allow providers who reduce health care costs for their patients to retain a percentage of the savings they generate.

Objective

To examine ACO and market factors associated with superior financial performance in Medicare ACO programs.

Methods

We obtained financial performance data from the Centers for Medicare and Medicaid Services (CMS); we derived market-level characteristics from Medicare claims; and we collected ACO characteristics from the National Survey of ACOs for 215 ACOs. We examined the association between ACO financial performance and ACO provider composition, leadership structure, beneficiary characteristics, risk bearing experience, quality and process improvement capabilities, physician performance management, market competition, CMS-assigned financial benchmark, and ACO contract start date. We examined two outcomes from Medicare ACOs’ first performance year: savings per Medicare beneficiary and earning shared savings payments (a dichotomous variable).

Results

When modeling the ACO ability to save and earn shared savings payments, we estimated positive regression coefficients for a greater proportion of primary care providers in the ACO, more practicing physicians on the governing board, physician leadership, active engagement in reducing hospital re-admissions, a greater proportion of disabled Medicare beneficiaries assigned to the ACO, financial incentives offered to physicians, a larger financial benchmark, and greater ACO market penetration. No characteristic of organizational structure was significantly associated with both outcomes of savings per beneficiary and likelihood of achieving shared savings. ACO prior experience with risk-bearing contracts was positively correlated with savings and significantly increased the likelihood of receiving shared savings payments.

Conclusions

In the first year performance is quite heterogeneous, yet organizational structure does not consistently predict performance. Organizations with large financial benchmarks at baseline have greater opportunities to achieve savings. Findings on prior risk bearing suggest that ACOs learn over time under risk-bearing contracts.

Implications

Given the lack of predictive power for organizational characteristics, CMS should continue to encourage diversity in organizational structures for ACO participants, and provide alternative funding and risk bearing mechanisms to continue to allow a diverse group of organizations to participate.

Level of evidence

III

Introduction

In recent years, there has been a growing interest in using shared savings as a new approach to the financing of health care, departing from fee-for-service payment arrangements. The 2010 Affordable Care Act introduced shared savings payment arrangement as a key component of the accountable care organization (ACO) model and the Centers for Medicare and Medicare Services (CMS) first tested it in the Medicare Physician Group Practice Demonstration before implementation in ACOs.^1,2 ACOs are intended to encourage coordination and cooperation among providers through financial incentives for high quality and lowered spending for a defined patient population. The ACO model was instituted through the Pioneer and the Medicare Shared Savings Program (MSSP) in 2012.³ In both cases, ACOs that meet performance standards on quality of care are eligible to share in generated savings as performance-based bonus payments.^4,5

There are fundamental differences between the Pioneer and MSSP models. While the MSSP was designed as a permanent program, the Pioneer program started earlier and was intended to evaluate alternative payment models and test care coordination, quality improvement, and healthcare cost reduction in organizations with risk-bearing experience. Moreover, the defined population for which ACOs in these two programs assume risk differs. Unlike MSSPs which accept responsibility for at least 5,000 Medicare fee-for-service beneficiaries, most Pioneer ACOs accept risk for at least three times as many beneficiaries (15,000). Therefore, Pioneer ACOs are larger organizations with more physicians and often a more diverse set of providers.⁶ MSSP organizations can operate under either a one-sided risk model, where they can be rewarded financially for lowering health care costs but are not liable for shared losses, or a two-sided model, where they are liable for shared losses if they overspend. The upside-risk-only option is not available to Pioneer ACOs which have five payment arrangement options with varying degrees of shared savings and losses across years. There is both greater risk inherent in the Pioneer program and greater potential for financial reward with higher sharing rates and higher payment caps.

Financial performance data reported by the CMS show that out of 367 Medicare ACOs in the Pioneer and MSSP programs with contract start dates between 2012 and 2014, 193 (53%) ACOs (including 20 Pioneer ACOs) collectively held spending at $1.17 billion below their total assigned spending benchmarks, while 174 (47%) collectively overspent by a total of $819 million in the first performance year.^7–9 Among ACOs realizing savings, 86 (45%) ACOs received a total of $316 million in bonuses. Despite this wide variation in performance and savings across ACOs, little is known about factors associated with success in the Medicare ACO programs. In creating the ACO programs, the CMS intentionally did not specify necessary organizational forms or necessary capabilities, in part because there is little evidence linking provider characteristics to success under new payment models. As a result, research has shown that Medicare ACO participants are diverse on many levels.^10–16 ACOs include academic medical centers, physician-hospital organizations, independent practice associations, regional and public hospitals, multispecialty group practices, integrated delivery systems, federally qualified health centers, critical access hospitals, combinations of all of the above organizational structures, or include none of these. The population of beneficiaries in these organizations is equally diverse in terms of demographics and comorbidity patterns.¹² It is not clear whether success in the Medicare ACO programs varies systematically according to organizational characteristics, beneficiary characteristics, ACO capabilities, or even market-level factors.

Prior literature on factors associated with financial performance of ACOs hinged on anecdotal evidence, pairwise correlations, qualitative interviews with ACO leadership for the MSSP program, simulation studies, and research focusing on the Pioneer program.^5,17–21 Probable factors suggested to influence ACO success include beneficiary characteristics and management (beneficiary turnover, high-need patient targeting, utilization of individual care plans, and beneficiary engagement);^5,17,18 the use of best practices with evidence-based care and electronic health records;¹⁹ historical spending; and geographical location.^5,20,21 However, there is a dearth of empirical research confirming or refuting these conjectures. Moreover, organizational structure as well as care management proficiency and clinical integration may influence performance in ACO programs. Understanding how these factors interact with financial performance could inform on what strategies could or could not be changed for current organizations and potential participants. Our research offers a comprehensive analysis of financial performance data from ACOs first performance year and examines organizational and regional market factors associated with success in achieving savings and earning shared savings payments in the MSSP and Pioneer programs.

Conceptual Model

We hypothesized that several organizational or market-level factors are associated with financial performance in the ACO programs. We expect infrastructure (e.g. size) to have an effect on the amount of savings per beneficiary and whether the ACO receives shared savings. Smaller ACOs may be more nimble and have greater flexibility to implement change and larger ACOs likely have more support and resources to do so and can spread the fixed costs of some investments (e.g. health technology) over a larger number of patients. However, mid-sized ACOs may be too large to make rapid changes and not have the resources to make productive capital investments: small and large ACO may have an advantage over medium size ACOs. Since primary care providers coordinate care, we expect that a greater proportion of such practitioners in the organization may increase the ACO’s ability to realize savings and earn bonuses. Also, empowering physicians in the decision making process with a greater proportion of practicing physicians on the ACO governing board and physician leadership may be favorable to achieving goals of enhancing performance and controlling cost, therefore facilitating savings. Offering financial incentives to physicians (a positive reinforcement) may be important in influencing practice and changing physician behavior so as to generate savings for the organization. ACOs more experienced with risk bearing contracts may be better able to establish core capabilities in coordinating care for the patients they serve²² and therefore may achieve greater savings.

We anticipate that a stronger emphasis on clinical care activities (e.g. greater quality improvement capabilities such as initiatives to reduce hospital re-admissions) and a higher spending history (i.e. higher financial benchmark) will increase the magnitude of savings and the likelihood of shared saving receipt. Although we anticipate that a larger population of beneficiaries with high-needs (e.g. minorities, disabled, and dual eligible patients) may create additional challenges for the ACO; a larger population of dual eligible beneficiaries, which has shown in an earlier program (i.e. the Physician Group Practice Demonstration) to increase savings,²³ could mean that there is more room for savings.

We also predict that greater integration in the regional market, which may promote less competition as the Medicare ACO programs expand,²⁴ will promote savings and the probability that the ACO gets shared savings payments.

Methods

We conducted a cross-sectional study examining Medicare ACOs’ financial performance in the first contract year. We described the relationship between ACO performance, size, structure, patient population characteristics, quality improvement capabilities, assigned financial benchmark, and market environment. We used a multivariate linear regression model to examine which ACO-level and regional market factors were associated with savings per beneficiary, our continuous outcome variable. We also used logistic regression to examine potential determinants of whether an ACO earned a shared savings payment from Medicare. For organizations which held spending below their assigned benchmark, shared savings payments are contingent on the ACO’s achieving savings in excess of a Minimum Savings Rate (MSR) defined by CMS. Multiple imputation methods²⁵ were used to incorporate one explanatory variable with missing values. Estimation combined all imputations and the standard errors were adjusted accordingly.

Data

We combined three data sources: 1) data from three waves of the National Survey of ACOs (NSACO), 2) market level factors from Medicare fee-for-service enrollees’ claims data from 2011 to 2013, and 3) financial performance and ACO descriptive data publicly available from CMS. The NSACO surveyed ACOs formed between 2012 and 2015: wave 1 surveyed organizations formed prior to September 2012, wave 2 surveyed ACOs formed between September 2012 and July 2013, and wave 3 surveyed ACOs formed between July 2013 and January 2015. ACOs were identified for the survey through publicly available studies, documents and announcements; with a cumulative response rate for Medicare ACOs of 69%. The most knowledgeable ACO-affiliated personnel (e.g. CEOs, executive director, and chief medical officers) responded to the survey either online or via telephone.¹³ The survey domains offer in depth information on organizational characteristics (e.g. participating provider organizations, governance and structure, contract characteristics, physician performance and compensation, payer mix), capabilities (e.g. care management and transition programs, patient activation and engagement programs, experience with risk bearing, performance monitoring, quality improvement and Health Information Technology infrastructure capabilities), and ACO activities (e.g. areas to reduce spending, new quality improvement initiatives, changing physician compensation models, shared savings distribution, clinical involvement in designing initiatives).

Our definition of a regional market area is a hospital referral region (HRR). The Dartmouth Atlas of Health Care defines 306 HRRs, each consisting of a set of hospital service areas grouped based on referral patterns to hospitals in the region. HRR measures are frequently used for geographic distribution of care delivery. Using Medicare claims data, we calculated competition and ACO penetration measures in the market.

Finally, CMS publically releases data for all Medicare ACOs on the number of assigned beneficiaries, personnel composition, and financial risk (track) model. Performance variables in the CMS data inform on whether the ACO successfully reported quality data, ACO benchmark spending, assigned beneficiary total expenditures, generated savings/losses, and earned shared savings/owed losses.

Study sample

We included all Medicare ACOs (MSSP and Pioneer participants) with reported financial information in the first contract year (for Pioneers January 2012–December 2012; for MSSPs, performance period varies by start date (either April 2012, July 2012, January 2013, or January 2014). Performance year is defined as a 12-month period starting on January 1 of each calendar year. However, for ACOs with start dates of April 1, 2012 and July 1, 2012, the first performance year is 21 months and 18 months respectively.²⁶

The first performance year data reported by CMS were available for 367 ACOs, and 248 of these have linked data from NSACO. We further matched ACOs to regional market areas (HRR). We excluded organizations that could not be matched to a defined HRR, resulting in exclusion of one ACO located in Puerto Rico. Finally, we only analyzed ACOs with no missing values in our explanatory variables: the sample reduced to 215 NSACO ACO responders.

Outcome measures

Outcome measures were obtained from the CMS reports on financial performance in year 1. Our first outcome (savings per beneficiary) was derived from gross savings. Gross savings, which can be positive or negative, measures the difference between the ACO’s financial benchmark and the total expenditures incurred for their attributed patients in the performance year.⁸ We divided gross savings by the number of assigned beneficiaries to generate a variable measuring savings per beneficiary.

For organizations that successfully hold spending below their expenditure targets, they will receive shared savings payments if the realized savings exceeds the MSR of a flat 2% for the two-sided risk MSSP and 1% for the Pioneer model from the cost benchmark. For ACOs under the one-sided risk model, the MSR varies from 2% to 3.9% from the financial benchmark. Conversely, under the two-sided risk model where organizations accept downside risk, ACOs which overspend above the Minimum Loss Rate of 1% to 2.7% for Pioneers or a flat 2% for MSSP under the two-sided risk model (i.e. Track 2 MSSP) are liable for shared losses.²⁷ Our sample included twenty Pioneers and four MSSPs in Track 2.

ACOs are categorized into four groups identifying whether they collected shared savings (N=55 of the 215 linked ACOs), whether they underspent but savings were not enough to receive bonuses (N=56 of 215), whether they overspent but did not have to repay Medicare (N=102 of 215), and whether they were financially liable to Medicare for overspending (N=2 of 215). We only used these categories to create our second outcome variable, whether the ACO received shared savings payments. On one hand we had the group of “savers” that received shared savings (N=55) and on the other hand, we combined the remaining three categories (N=160) to define the group that did not receive shared savings by either not saving enough or by overspending.

There are limits on the percent of savings that an ACO can earn: MSSPs earn shared savings of up to 50% of savings generated under the one-sided model and up to 60% under the two-sided model of sharable savings, and Pioneers can earn up to 70%, dependent on performance and reporting of the quality measures. In the first performance year, ACOs qualified for shared savings for completely and accurately reporting quality measures, often called pay for reporting.²⁷ Hence, the group of ACOs in our sample which realized savings without bonuses from Medicare also includes four ACOs eligible for bonuses but which failed to report quality.

ACO Attributes and Market-level Characteristics

Explanatory variables were divided into nine conceptual groups including size and composition, organizational structure, beneficiaries’ characteristics, risk bearing experience, quality and process improvement capabilities, physician performance management, market competition, benchmark spending, and ACO start date.

From the CMS data we assessed ACO size and composition using two variables, the proportion of participating medical personnel offering primary care services in the organization and the number of full-time-equivalent participating medical personnel. Using the NSACO, ACO structure was characterized by an indicator for whether physicians led the organization (as opposed to hospital leadership, joint hospital-physician leadership, or other leadership coalitions) and a continuous variable measuring proportion of practicing physicians in the governing board. Using CMS data, we included three measures of beneficiaries’ characteristics including the proportions of dual-eligible beneficiaries (those who qualify for both Medicare and Medicaid), disabled beneficiaries, and minority beneficiaries. Notably, CMS only reported size/composition and beneficiaries’ characteristics for MSSP organizations (not Pioneers); these measures were therefore omitted when the full sample was analyzed.

Measures of risk-bearing and quality improvement capabilities were obtained from the NSACO. We included information on risk-bearing status through a dichotomous variable for whether the ACO or provider group had prior experience with risk-bearing contracts. Quality and process improvement capabilities were assessed using a variable identifying the extent to which the organization measured or implemented systems to reduce preventable hospital re-admissions. This variable was measured on a 1 to 9 scale provided in the survey item with 9 referring to the ACO reporting having a fully developed program and being actively engaged in reducing hospital re-admissions. An indicator, from NSACO, measuring whether the ACO offered individual financial incentives to physicians as a way to manage physician performance was included to capture factors affecting physicians’ motivation. All variables drawn from the NSACO were available in all three waves and therefore all our data are reflective of the ACOs in the first contract year.

Although some ACOs may span more than one HRR, each ACO was assigned a single HRR, the designated market area, based on its headquarter location for simplicity. In 86% of the cases, the HRR associated with the headquarter location matched the ACO service area. We included two market-level measures calculated using Medicare claims data. The first measured competition, defined with a Herfindahl–Hirschman Index (HHI) in the HRR of an ACO. The HHIs were calculated based on concentration of allowed Medicare charges across all provider practices in 2011 to avoid concerns about potential endogeneity. The HHIs specify whether care in the market is concentrated in select practices—defined as tax identification numbers—or shared equitably among many practices.¹⁰ HHI in the extreme case of a monopoly (a single provider in the market) would be equal to 10,000 (the square of 100% market share). HHI can also be very small when market share is spread proportionately across many providers. Thus, market concentration informs on whether the market is competitive (HHI<1,500), moderately concentrated (1,500≤HHI<2,500), or highly concentrated (HHI≥2,500).²⁸ Using 2012 and 2013 Medicare claims data, our second market-level factor measured the degree of ACO penetration, calculated as the proportion of Medicare fee-for-service beneficiaries in a given HRR that are attributed to ACOs in the first performance year. A low market penetration measure indicates that a small number of beneficiaries were attributed within the HRR and that more beneficiaries could be attributed: hence, there could be greater potential for an expansion of the Medicare ACO programs to cover these unattributed Medicare beneficiaries.

We segmented the distribution of financial benchmarks (assigned by CMS) per beneficiary for ACOs in our sample into three tertiles and created an indicator variable for whether the organization’s benchmark per beneficiary is situated in the top tertile. Our study investigated five ACO cohorts: 1) Pioneers with start date 01/01/2012 (N=20 of 215), and MSSPs with start dates 2) 04/01/2012 (N=21); 3) 07/01/2012 (N=62); 4) 01/01/2013 (N=59); and 5) 01/01/2014 (N=53). We included five dichotomous variables for each of these ACO cohorts to control for effects that are specific to these cohorts in the first contract year.

Results

Descriptive statistics

Table 1 provides descriptive characteristics for our sample. ACOs tended to be staffed primarily by primary care providers, were mostly led by physicians, and had the majority of their board members being practicing physicians. About forty percent of ACOs offered financial incentives to their network physicians as an approach in motivating physician performance. The proportion of historically high-spending and high-needs assigned beneficiaries was non-trivial but relatively lower than in the general population of Medicare beneficiaries: while dual eligible, disabled, and minority beneficiaries represented 8% to 16% on average of ACOs’ assigned beneficiaries in our sample, the proportion of 2012 Medicare beneficiaries with these characteristics varied from 19% to 26% on average.

Table 1.

Descriptive Statistics of Variables in Performance Year 1

VARIABLES	Data Source	Mean	Std. Dev.
Savings per Beneficiary ($)	Financial Reports-CMS	55.58	765.89
Receive Shared Savings (=1 if yes)	Financial Reports-CMS	0.26	0.44
Size/Composition*
Proportion of Participating PCPs and NPs†	Financial Reports-CMS	0.54	0.22
Number of Participating FTE PCPs, NPs, SPECs, PAs, CNSs‡	Financial Reports-CMS	481.49	663.38
Structure
Physician Leadership (=1 if yes)	NSACO	0.62	0.49
Proportion of the governing board are practicing physicians	NSACO	0.68	0.24
Beneficiaries
Proportion of Dual Eligible Beneficiaries	Financial Reports-CMS	0.08	0.11
Proportion of Disabled Beneficiaries	Financial Reports-CMS	0.15	0.08
Proportion of Minority Beneficiaries	Financial Reports-CMS	0.16	0.14
Risk+
ACO or provider group has previous experience with risk-bearing contracts (=1 if yes)	NSACO	0.59	0.49
Quality and Process Improvement Capabilities
ACO engaged in reducing preventable hospital re- admissions (1–9 scale)	NSACO	6.42	1.50
Physician Performance Management
Individual Financial Incentives to Physicians (=1 if yes)	NSACO	0.41	0.49
Competition
HHI (Market Concentration Index at the HRR Level)	Medicare Claims	294.35	543.40
Market Penetration at the HRR level	Medicare Claims	0.34	0.25
Benchmark
Financial Benchmark per Beneficiary, Top tertile ($12,141 to $21,981; =1 if yes)	Financial Reports-CMS	0.33	0.47
Cohort
Indicator for Start Period = 1 (01/01/2012) (Pioneer)	Financial Reports-CMS	0.09	0.29
Indicator for Start Period = 2 (04/01/2012)	Financial Reports-CMS	0.10	0.30
Indicator for Start Period = 3 (07/01/2012)	Financial Reports-CMS	0.29	0.45
Indicator for Start Period = 4 (01/01/2013)	Financial Reports-CMS	0.27	0.45
Indicator for Start Period = 5 (01/01/2014)	Financial Reports-CMS	0.25	0.43

N=215.

^*Measures of Size/Composition and Beneficiaries Characteristics are only available for MSSP ACOs (195 Organizations).

⁺Risk Bearing Status is available for 170 ACOs.

^†PCPs: Primary Care Physicians and NPs: Nurse Practitioners.

^‡FTE PCPs, NPs, SPECs, PAs, CNSs: Full Time Equivalent Primary Care Physicians, Nurse Practitioners, Specialists, Physician Assistants, and Clinical Nurse Specialists.

More than half of ACOs had prior experience with risk-bearing contracts (e.g. capitation). Although there were missing information in this variable for 21% of ACOs, there was no evidence for a specific pattern in nonresponse. In order to preserve non-missing information in all the other covariates in the analysis, we employed multiple imputations for this variable only in the empirical analysis. The majority of organizations attested to taking considerable measures and effort in reducing preventable hospital re-admissions (47%). Financial benchmark per beneficiary ranged from $11,364 to $21,981 with the top tertile being $12,141 to $21,981.

Market concentration ranging from 16.1 to 6133.7 in our sample was relatively low on average (294.35), indicating that ACOs faced considerable competition which could in theory imply lower prices in commercial contracts, possibly higher quality of care and innovation, and lower degrees of care coordination.²⁴ In fact, only 11 ACOs operated in moderately concentrated markets and 4 organizations were in highly concentrated markets. There was also considerable variation in the penetration measure across markets.

Empirical Analysis

Regression results from the models in the Appendix are reported in Tables 2 and 3. Table 2 presents estimation results of multivariate regressions for the entire sample including both Pioneer and MSSP ACOs. When including measures of size/composition and beneficiaries characteristics, we further restricted the sample to MSSP ACOs only (Table 3).

Table 2.

Regression Results-Determinants of Savings and Shared Saving Payments in First Performance Year

	(1)	(2)	(3)	(4)
VARIABLES	Savings per Beneficiary		P(Shared Savings Payments)
	Coeff	Std. Err	Coeff	Std. Err
Structure
Physician leadership	24.14	(124.87)	−0.04	(0.40)
Proportion of the governing board are practicing physicians	195.99	(264.24)	0.90	(0.89)
Risk
ACO or provider group has previous experience with risk-bearing contracts	193.01	(129.55)	1.04***	(0.41)
Quality and Process Improvement Capabilities
ACO engaged in reducing preventable hospital re- admissions	34.88	(36.83)	0.15	(0.12)
Physician Performance Management
Individual financial incentives to physicians	78.52	(113.45)	−0.20	(0.36)
Competition
HHI (Concentration in the HRR market)	−0.02	(0.10)	−0.0005	(0.001)
Market penetration in the HRR market	3.67	(216.51)	−0.18	(0.74)
Benchmark
Financial benchmark per beneficiary, top tertile	127.10	(132.42)	0.64	(0.40)
Cohort
Indicator for start period = 2 (04/01/2012)	107.86	(258.81)	−0.51	(0.75)
Indicator for start period = 3 (07/01/2012)	−138.32	(212.66)	−0.45	(0.61)
Indicator for start period = 4 (01/01/2013)	−8.50	(210.56)	−0.82	(0.62)
Indicator for start period = 5 (01/01/2014)	18.29	(215.89)	−0.57	(0.63)
Observations	215		215

Notes: The estimations result from multiple imputations of the risk-bearing status measure. Robust standard errors in parentheses.

^***p<0.01

Table 3.

Regression Results-Determinants of Savings and Shared Saving Payments in First Performance Year (MSSP only)

	(1)	(2)	(3)	(4)
VARIABLES	Savings per Beneficiary		P(Shared Savings Payments)
	Coeff	Std. Err	Coeff	Std. Err
Size/Composition
Proportion of PCPs and NPs	366.31	(311.94)	1.25	(0.97)
Number of FTE PCPs, NPs, SPECs, PAs, CNSs	−0.03	(0.11)	0.0001	(0.0003)
Structure
Physician leadership	−2.77	(143.79)	0.10	(0.47)
Proportion of the governing board are practicing physicians	306.02	(310.61)	1.56	(1.19)
Beneficiaries
Proportion of dual eligible beneficiaries	−624.02	(829.65)	−0.44	(2.90)
Proportion of disabled beneficiaries	529.63	(840.68)	1.31	(2.89)
Proportion of minority beneficiaries	−142.06	(638.25)	−2.35	(2.42)
Risk
ACO or provider group has previous experience with risk-bearing contracts	196.06	(139.88)	1.09***	(0.42)
Quality and Process Improvement Capabilities
ACO engaged in reducing preventable hospital re- admissions	40.49	(41.18)	0.19	(0.14)
Physician Performance Management
Individual financial incentives to physicians	53.78	(126.07)	−0.37	(0.40)
Competition
HHI (Concentration in the HRR market)	−0.06	(0.17)	−0.001	(0.001)
Market penetration in the HRR market	54.31	(251.75)	0.06	(0.85)
Benchmark
Financial benchmark per beneficiary, top tertile	118.32	(154.69)	0.62	(0.49)
Cohort
Indicator for start period = 3 (07/01/2012)	−283.52	(210.12)	−0.03	(0.61)
Indicator for start period = 4 (01/01/2013)	−142.24	(226.92)	−0.49	(0.69)
Indicator for start period = 5 (01/01/2014)	−124.69	(227.05)	−0.02	(0.70)
Observations	194		194

Notes: The estimations result from multiple imputations of the risk-bearing status measure. Robust standard errors in parentheses.

^***p<0.01

As shown in Table 2 Column 1, although not statistically significant, ACO-level factors for which we estimated positive coefficients in the model with savings per beneficiary as the outcome variable include physician leadership in the ACO; proportion of physicians in the governing board; prior experience with risk-bearing contact; the organization’s ability to employ measures in reducing preventable hospital re-admissions and offer financial incentives to their physicians; and being identified as an organization with high spending at baseline (top tertile for financial benchmarks). The coefficient estimate on market penetration was also positive but the relationship with savings per beneficiary was imprecisely estimated. We found that local market concentration had no correlation with the organization’s ability to save. Nonetheless, this is consistent with a vast majority of ACOs (93%) located in competitive markets (HHI<1,500).

We present results on our second outcome variable, whether the ACO earned bonuses payments, in Table 2 Column 3 for the entire sample. We found that previous experience with risk-bearing contracts significantly increased the likelihood of earning bonus payments. Calculation of odds ratios suggest that prior risk experience increases the odds of earning shared savings payments by a factor of about three.

Since ACO size/composition and beneficiary characteristics are only reported for the subset of MSSP organizations, we further restricted the sample to the group of MSSP ACOs and estimated our multivariate regression models. This exercise also allowed us to examine any heterogeneity in the associations between the covariates and performance outcomes that could result from organizational differences in the Pioneer and MSSP programs.⁶ Regression results are presented in Table 3. The relationships estimated in Table 2 held in general and additional covariates with positive regression coefficients—though the coefficients were not statistically significant—on financial performance included the proportion of primary care providers in the organization and the proportion of assigned disabled Medicare beneficiaries. ACOs with a higher proportion of dual eligible and minority Medicare beneficiaries were correlated with less savings. Similarly, prior risk experience had a positive and statistically significant association with the probability of bonus payment (Table 3 Column 3), increasing the odds by a factor of about three.

In regressions not shown, we also tested for the association between beneficiary turnover and ACO’s financial performance. Beneficiary turnover may negatively affect accountability and performance.^17,29 We used Medicare claims data from 2012 to 2013 to construct a measure of beneficiary turnover/stability across ACOs using the Medicare methodology for attribution of Medicare beneficiaries to ACOs.³⁰ We found that although the regression coefficient associated with beneficiary stability was positive on the ACO’s ability to realize savings and receive bonus payments beneficiary stability was not predictive of success in the ACO programs. Since using this variable required exclusion of ACOs with contract start dates in 2014, limiting the sample size even further, we did not include it in the baseline model. Although not reported, we also run our baseline models with all 365 ACOs with defined HRRs and a small subset of covariates and results were similar to those estimated using our analytic sample.

Test for Heterogeneous Associations with Savings per Beneficiary

Considering the distribution of savings per beneficiary, the standard deviation in Table 1 suggests considerable variation across ACOs. Figure 1 confirms such a variation. Consequently, one may want to examine variation in the associations between savings per beneficiary and the different covariates at different points of this distribution. We opted for quantile regressions and re-estimated our baseline regression model. The distribution of savings per beneficiary was divided into four (quartiles) with the first quartile representing the 25^th percentile of the distribution, the second quartile being the median or the 50^th percentile, and the third quartile identifying the 75^th percentile. Results using our sample of 215 ACOs (Table 4) suggest that physician leadership may matter most early in the distribution of savings per beneficiary, with a marginally significant association at the 25^th percentile (an increase of $128), which fades away along the distribution. We also found prior experience with risk bearing contracts to be significantly associated with savings per beneficiary at the 75^th percentile (an increase of $270). Although not statistically significant in Table 2, the association between financial benchmark and savings per beneficiary was estimated to be statistically significant and monotonic in Table 4. Classification in the top tertile for financial benchmark per beneficiary was associated with $570 more savings per beneficiary at the 75^th percentile. We estimated similar associations for the MSSP-only sample and therefore do no report these results.

Figure 1.

Distribution of Savings per Beneficiary in First Performance Year

Table 4.

Quantile Regression Results-Determinants of Savings in First Performance Year

	(1)	(2)	(3)	(4)	(5)	(6)
VARIABLES	25th Percentile		50th Percentile		75th Percentile
	Coeff	Std. Err	Coeff	Std. Err	Coeff	Std. Err
Structure
Physician leadership	128.46*	(76.47)	40.17	(106.74)	6.44	(134.74)
Proportion of the governing board are practicing physicians	−107.68	(161.73)	−95.88	(224.19)	159.98	(272.90)
Risk
ACO or provider group has previous experience with risk-bearing contracts	−0.42	(70.96)	151.58	(106.03)	270.12**	(114.97)
Quality and Process Improvement Capabilities
ACO engaged in reducing preventable hospital re-admissions	10.41	(22.36)	−5.85	(31.59)	6.41	(36.53)
Physician Performance Management
Individual financial incentives to physicians	52.11	(69.29)	−5.33	(97.27)	−68.28	(115.32)
Competition
HHI (Concentration in the HRR market)	0.01	(0.06)	−0.004	(0.09)	−0.05	(0.10)
Market penetration in the HRR market	83.18	(132.76)	−89.05	(182.50)	−258.86	(213.57)
Benchmark
Financial benchmark per beneficiary, top tertile	171.21**	(80.97)	394.25***	(126.80)	569.72***	(131.89)
Cohort
Indicator for start period = 2 (04/01/2012)	−183.21	(157.39)	−124.68	(225.30)	−75.85	(269.95)
Indicator for start period = 3 (07/01/2012)	−164.89	(129.37)	4.82	(187.35)	−55.05	(216.50)
Indicator for start period = 4 (01/01/2013)	−3.67	(128.22)	80.48	(177.02)	−38.79	(204.72)
Indicator for start period = 5 (01/01/2014)	12.26	(131.28)	46.93	(183.26)	−90.80	(207.02)
Observations	215		215		215

Notes: The estimations result from multiple imputations of the risk-bearing status measure. Robust standard errors in parentheses.

^***p<0.01,

^**p<0.05,

^*p<0.1

Discussion

Financial performance data in year 1 of the Medicare ACO programs and the proportion of ACOs which overspent (48% in our sample) somewhat confirm earlier predictions of likely financial loss in the early years of the program.³¹ We observe rich variation across ACOs in terms of beneficiary population, quality improvement capabilities, and organizational composition. While results should not be interpreted as causal relationships, our findings suggest that no particular characteristic of organizational structure and composition was significantly associated with both realization of savings and likelihood of earning shared savings. Although ACO financial performance is heterogeneous, the directions and strengths of the relationships estimated do not indicate consistent heterogeneity across organizational structures, compositions, and levels of development. In the first performance year, many different types of ACOs with varying levels capabilities, were able to reduce spending. ACOs of all compositional types were equally likely to achieve savings—or fail to achieve savings. Nonetheless, the positive coefficient estimates found on capabilities provide weak evidence that capabilities may matter but we only examined one set of capabilities among many.

Among the ACO attributes used in this analysis, the positive and significant association between risk bearing and financial performance may suggest that ACOs with such experiences have acquired knowledge and strategies on how to better operate in a risk-based environment. This relationship may strengthen even more with organizations transitioning into the two-sided risk model over time, since models where ACOs operate under a risk-based arrangements have the potential to induce more meaningful systematic change due to the degree of financial risk involved.²⁶ Such expectations are consistent with spending trajectories found in payment reforms involving downside risk (i.e. The Blue Cross Blue Shield Massachusetts Alternative Quality Contract-AQC).³² Alternatively, voluntary selection into downside risk can raise concerns for selection bias because ACO (risk-model) participation is voluntary, and further research would be needed to disentangle these two effects. Nonetheless, transition from Track 1 (upside financial risk only) to Track 2 may take longer to materialize as CMS has allowed continuation in Track 1 for a second (three-year) contract period to retain ACOs in the program.³³

The positive association between size of Medicare-assigned financial benchmark established based on historical spending patterns, larger savings per beneficiary, and a higher likelihood of receiving shared savings provides some evidence for early exploratory work and claims that being located in high cost regions increases the propensity of an ACO to achieve savings.²¹ Considering statistical significance of the estimates at different points of the distribution for savings per beneficiary, these results may indicate that organizations with initial larger benchmarks may have had more waste at baseline and therefore more room for improvement than their more efficient counterparts with smaller assigned benchmarks. Although greater market penetration was positively correlated with savings and the probability of earning bonus payments, there was no apparent correlation between greater market concentration, spending, and the likelihood of receiving shared savings payments. As the Medicare ACO programs expand and there is increased learning from past experiences, we may expect a stronger association between market concentration and financial outcomes.

This analysis uses unique and robust survey data on ACOs and accounts for many potential confounders. The use of many cohorts of the Medicare ACO programs offers additional robustness to our analyses and contribution. Nonetheless, there are limitations to our analysis. The empirical analysis is conducted with only ACOs linked to the survey database (248 ACOs) 68% of the 367 ACOs for which financial performance data were reported by the CMS. The sample is further reduced with missing values in some of our covariates. Testing for differences in outcome variables and covariates drawn from the CMS data (size/composition, beneficiary characteristics, and benchmark measures) between our sample of complete cases and the rest of ACOs not analyzed (N=152), most characteristics were not statistically different, with a few exceptions (Appendix, Table A1). Besides the gain of statistical power that a greater sample size would add to the estimation results, relatively greater average proportions of primary care providers and minority beneficiaries in the sample not used may well imply that the coefficients estimated are lower bound estimates (keeping all other factors constant). Moreover, we are studying only the first performance year in the program. Different patterns may emerge if some groups are better able to make changes over time. A longitudinal analysis with additional performance years’ data could help identify persistent and robust associations. Finally, we are using Medicare-reported performance data in these analyses which rely on a specific payment methodology that makes it easier for traditionally high-spending ACOs to achieve savings.³⁴ Further research should investigate the legitimacy of the savings calculated by the Medicare program.

Starting with 114 MSSP ACOs in 2012 and 32 Pioneers, the count has inflated over the years to reach 433 MSSPs by mid-2016. The programs have witnessed changes in payment methodology³⁵ and quality measures. Since the end of the first performance year, six ACOs have changed from the Pioneer program to the MSSP, 13 ACOs left the programs, and Medicare has recently announced 21 participants in the Next Generation ACO program.³⁶ Our results are, therefore, informative for existing ACOs but also have implications for organizations considering accountable care. They should understand that prior experience with risk bearing contracts, although not indispensable, may be beneficial for success in this environment. Efficient organizations, who have reduced growth in health care spending for their assigned patients prior to ACO implementation, may find it difficult to further reduce spending while participating. Physician engagement in leadership and through financial incentives may help organizations improve financial performance. In light of our results, CMS should continue to encourage diversity in organizational structures for ACO participants, and provide alternative funding and risk bearing mechanisms to continue to allow a diverse group of organizations to participate.

Appendix

Statistical Analysis

We used a linear regression and a logistic model to determine the association between organizational characteristics, market environment factors, and financial performance in the first contract year. The ACO’s ability to realize savings was estimated in the model as follows:

$${\text{SavingsPerBene}}_{\mathrm{i}\mathrm{l}\mathrm{c}}={\mathrm{\alpha }}_0+\mathrm{\beta }{\mathrm{X}}_{\mathrm{i}\mathrm{l}\mathrm{c}}+{\mathrm{\theta }}_{\mathrm{c}}+{\mathrm{\epsilon }}_{\mathrm{i}\mathrm{l}\mathrm{c}}$$ (1)

The probability of earning shared savings payments was specified in equation 2:

$$\mathrm{P}{(\text{ReceiveSharedSavings})}_{\mathrm{i}\mathrm{l}\mathrm{c}}=\mathrm{f}({\mathrm{X}}_{\mathrm{i}\mathrm{l}\mathrm{c}},{\mathrm{\theta }}_{\mathrm{c}})$$ (2)

Where indicators ilc refer to ACO i in regional market l in cohort c. The vector X represents organization-specific attributes and market-level characteristics described above, and represents the cohort of the ACO.

Table A1.

Financial Performance and Organization Characteristics of Medicare ACOs in the First Performance Year: Complete cases linked to the National Survey of Accountable Care Organization versus CMS Medicare ACOs not used

VARIABLES	Authors Complete Cases		CMS Medicare ACOs not used—Incomplete Cases and ACOs not linked to the NSACO		P value
	(n=215)		(n=152)
Savings per Beneficiary	50.87	(762.84)	42.79	(564.82)	0.91
Receive Shared Savings	0.25	(0.43)	0.20	(0.40)	0.29
Size/Compositiona
Proportion of PCPs and NPs	0.54	(0.22)	0.58	(0.23)	0.07
Number of FTE PCPs, NPs, SPECs, PAs, CNSs	479.76	(662.11)	392.96	(590.17)	0.22
Beneficiariesa
Proportion of Dual Eligible Beneficiaries	0.08	(0.11)	0.09	(0.11)	0.81
Proportion of Disabled Beneficiaries	0.15	(0.08)	0.15	(0.08)	0.80
Proportion of Minority Beneficiaries	0.16	(0.14)	0.19	(0.16)	0.06
Benchmark
Financial Benchmark per Beneficiary ACO, top tertile	0.35	(0.48)	0.30	(0.46)	0.31

Notes: All variables presented as mean (standard deviation) with P value by t-test. Data are publicly available from the Centers for Medicare and Medicaid Services performance year 1 financial reports for ACOs.

^aSize/composition and beneficiary variables are only reported by CMS for the 335 MSSP participants (195 complete cases and 140 Incomplete cases).