Changes in Patients’ Experiences in Medicare Accountable Care Organizations

Abstract

BACKGROUND

Incentives for accountable care organizations (ACOs) to limit health care use and improve quality may enhance or hurt patients’ experiences with care.

METHODS

Using Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey data covering 3 years before and 1 year after the start of Medicare ACO contracts in 2012 as well as linked Medicare claims, we compared patients’ experiences in a group of 32,334 fee-for-service beneficiaries attributed to ACOs (ACO group) with those in a group of 251,593 beneficiaries attributed to other providers (control group), before and after the start of ACO contracts. We used linear regression and a difference-in-differences analysis to estimate changes in patients’ experiences in the ACO group that differed from concurrent changes in the control group, with adjustment for the sociodemographic and clinical characteristics of the patients.

RESULTS

After ACO contracts began, patients’ reports of timely access to care and their primary physicians’ being informed about specialty care differentially improved in the ACO group, as compared with the control group (P = 0.01 and P = 0.006, respectively), whereas patients’ ratings of physicians, interactions with physicians, and overall care did not differentially change. Among patients with multiple chronic conditions and high predicted Medicare spending, overall ratings of care differentially improved in the ACO group as compared with the control group (P = 0.02). Differential improvements in timely access to care and overall ratings were equivalent to moving from average performance among ACOs to the 86th to 98th percentile (timely access to care) and to the 82nd to 96th percentile (overall ratings) and were robust to adjustment for group differences in trends during the preintervention period.

CONCLUSIONS

In the first year, ACO contracts were associated with meaningful improvements in some measures of patients’ experience and with unchanged performance in others. (Funded by the National Institute on Aging and others.)

In the Medicare accountable care organization (ACO) programs, participating provider groups are rewarded financially for limiting the use of health care and improving the quality of care. Specifically, ACOs that achieve spending levels sufficiently lower than targets set by Medicare are eligible to receive a share of the savings.^1,2 If spending exceeds the target, some ACOs — those in the Pioneer program and a few in the Medicare Shared Savings Program (MSSP) — must return a proportion of the excess to Medicare.

The proportion of savings or losses that accrues to an ACO depends on its performance on a set of quality measures, with measures of patients’ experiences from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey contributing heavily (25%) to the overall quality score of an ACO.^3–5 Rewarding ACOs for improving patients’ experiences encourages patient-centered care and may help mitigate incentives to limit important care that are inherent in ACO-like risk contracts. For ACOs and Medicare, it is also important to preserve or enhance patients’ experiences because beneficiaries who are dissatisfied with care in ACOs can seek care elsewhere, possibly causing ACOs to lose market share and leave the voluntary ACO programs. Thus, patient-reported experiences may provide key information for detecting unintended consequences of the ACO programs, gauging their viability, and assessing their overall effect on health care value. To our knowledge, the effects of ACO incentives on patients’ experiences have not been described previously.

Using CAHPS survey data from the period of 2010 through 2013 and linked Medicare claims, we compared experiences of care reported by Medicare beneficiaries served by provider organizations entering the ACO programs in 2012 with the experiences reported by beneficiaries served by other providers, before versus after the start of ACO contracts.

METHODS

STUDY DATA, PERIOD, AND POPULATION

The fee-for-service Medicare CAHPS survey is administered annually to a nationally representative, cross-sectional sample of traditional fee-for-service Medicare beneficiaries.^5–7 Because the survey is administered early in the year and assesses patients’ experiences with care in the prior 6 months, we defined the preintervention period to include survey data from 2010 through 2012 and the postintervention period to include 2013 survey data (Fig. 1). The 2013 survey, which over-sampled beneficiaries served by ACOs in 2012, was administered 8 to 14 months after MSSP ACO contracts started in April or July of 2012 and 14 to 17 months after Pioneer ACO contracts started in January 2012.

Figure 1. Survey and Study Periods Relative to the Start of Accountable Care Organization (ACO) Contracts in 2012.

The Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey of fee-for-service Medicare beneficiaries is administered annually from March through May; in the period from 2010 through 2013, a total of 82.2% of the respondents were surveyed by the end of April. The 2013 survey was administered 8 to 14 months after Medicare Shared Savings Program (MSSP) ACO contracts started in April or July of 2012 (the first and second rounds, respectively) and 14 to 17 months after Pioneer ACO contracts started in January 2012. The 2012 survey was administered an average of 3 months after the start of Pioneer ACO contracts, but the reference period included up to 4 months before the start dates because the CAHPS questions on patients’ experiences refer to the preceding 6 months of care. We did not expect substantial changes in patients’ experiences within 3 months after exposure to ACO contract incentives, and the 2012 survey reference period overlapped minimally with the MSSP ACO contract periods. Accordingly, we defined the preintervention period to include survey data from the period from 2010 through 2012 and the postintervention period to include 2013 survey data.

Using beneficiaries’ health insurance claim identification numbers, we linked each annual cross-sectional CAHPS sample from 2010 through 2013 to Medicare claims and enrollment data for the year before the start of the reference period for each survey (e.g., respondents to the 2013 survey who reported experiences with care received from late 2012 through early 2013 were linked to 2011 claims). We excluded beneficiaries who had no primary care services in linked claims and who therefore could not be attributed to an ACO or non-ACO provider group. We also excluded Medicaid enrollees because they were not sampled in the 2010 or 2011 survey. Results from a sensitivity analysis including Medicaid enrollees from the 2012 and 2013 surveys were similar to those we report. Finally, we excluded respondents who did not answer at least one question about their experiences as a patient (5.1% of the study population).

study VARIABLES

Comparison Groups

For each of the 32 Pioneer ACOs and 114 MSSP ACOs entering contracts with Medicare in 2012 and each of the 105 MSSP ACOs entering contracts in 2013,^8–11 we matched names of participating physicians and provider groups posted by the Centers for Medicare and Medicaid Services (CMS) or ACOs to National Provider Identifiers (NPIs) or taxpayer identification numbers (TINs), using publicly accessible databases (95% of physician and group names were matched to an NPI or TIN).^12–16 To limit misclassification due to physician turnover within constituent practices of ACOs over the study period, we defined ACOs as collections of TINs by converting participating NPIs to the primary TINs under which they billed claims in 2011. Results were similar in a sensitivity analysis that defined ACOs as groups of NPIs (see the Supplementary Appendix, available with the full text of this article at NEJM.org).

Following MSSP rules,² we assigned each CAHPS respondent in each survey to the ACO or non-ACO TIN that accounted for the most allowed charges for primary care services received by the respondent during the year of linked claims (see the Supplementary Appendix). We used claims preceding each survey period, rather than during each survey period, to better align assignments with lists of prospectively or preliminarily assigned beneficiaries supplied to ACOs by CMS to support population health management^2,17 and to minimize potential bias from ACO incentives to attract healthy patients in the postintervention period. We classified respondents assigned to provider groups that entered the Medicare ACO programs in 2012 as the ACO group and all other respondents meeting inclusion criteria as the control group.

Measures of Patients’ Experiences

We analyzed patients’ ratings of care in four domains: overall ratings of care and physicians, timely access to care, interactions with the primary physician, and care coordination and management (Table 1). We rescaled ratings to a consistent 0-to-10 scale and calculated composite scores for two domains composed of closely related items — timely access to care and interactions with the primary physician (see the Supplementary Appendix).

Table 1.

CAHPS Survey Measures of Patients’ Experiences with Care, According to Domain.^*

Survey Item	Survey Question	Original Scale†	Survey Years
Overall ratings
Rating of health care	What number would you use to rate all your health care in the past 6 months?	0–10	2010–2013
Rating of primary physician	What number would you use to rate your personal doctor?	0–10	2010–2013
Rating of specialist	What number would you use to rate the specialist you saw most often in the past 6 months?	0–10	2010–2013
Timely access to care
Timely access to urgent care	In the past 6 months, when you needed care right away, how often did you get care as soon as you thought you needed it?	1–4	2010–2013
Timely access to nonurgent care	In the past 6 months, not counting the times you needed care right away, how often did you get an appointment for your health care at a doctor’s office or clinic as soon as you thought you needed it?	1–4	2010–2013
Timely access to specialty care	In the past 6 months, how often was it easy to get appointments with specialists?	1–4	2010–2013
Interactions with primary physician
Clear communication	In the past 6 months, how often did your personal doctor explain things in a way that was easy to understand?	1–4	2010–2013
Careful listening	In the past 6 months, how often did your personal doctor listen carefully to you?	1–4	2010–2013
Respect	In the past 6 months, how often did your personal doctor show respect for what you had to say?	1–4	2010–2013
Sufficient time	In the past 6 months, how often did your personal doctor spend enough time with you?	1–4	2010–2013
Care coordination and management
Primary physician informed about specialty care	In the past 6 months, how often did your personal doctor seem informed and up to date about the care you got from specialists?	1–4	2011–2013
Patient care information available to primary physician	In the past 6 months, when you visited your personal doctor for a scheduled appointment, how often did he or she have your medical records or other information about your care?	1–4	2012 and 2013
Communication of test results	In the past 6 months, when your personal doctor ordered a blood test, x-ray, or other test for you, how often did someone from your personal doctor’s office follow up to give you those results?	1–4	2012 and 2013
Timely communication of test results	In the past 6 months, when your personal doctor ordered a blood test, x-ray, or other test for you, how often did you get those results as soon as you needed them?	1–4	2012 and 2013
Medication reconciliation	In the past 6 months, how often did you and your personal doctor talk about all the prescription medicines you were taking?	1–4	2012 and 2013
Patients’ access to visit notes	Visit notes sum up what was talked about on a visit to a doctor’s office. Visit notes may be available on paper, on a website, or by e-mail. In the past 6 months, did anyone in your personal doctor’s office offer you visit notes?	Yes or no	2012 and 2013

^*CAHPS denotes Consumer Assessment of Healthcare Providers and Systems.

^†For numeric scores, scales range from 0 (worst) to 10 (best) or from 1 (never) to 4 (always).

We expected that ACOs would have limited ability to improve patients’ ratings of physicians and physicians’ interpersonal skills, at least initially,¹⁸ but could achieve early gains in other measures of patients’ experiences by implementing patient-centered care management, access, referral, and information systems. We also expected that ACO incentives to limit utilization could have negative effects in several domains, including overall ratings of physicians and care, timely access to care, and interactions with physicians.

Covariates

From linked claims and enrollment data for each respondent, we assessed age, sex, race or ethnic group,^{19, 20} whether disability was the original reason for Medicare eligibility, and whether the respondent had end-stage renal disease or any of 27 conditions in the Chronic Conditions Data Warehouse (CCW). The CCW draws from diagnoses since 1999 to describe beneficiaries’ accumulated disease burden.²¹ From the year of the linked claims, we also calculated Hierarchical Condition Category (HCC) risk scores predicting future Medicare spending for each respondent.²² HCC risk scores are derived from demographic and diagnostic data in Medicare enrollment and claims files, with higher scores indicating higher predicted Medicare spending. (In our study, HCC risk scores ranged from 0.12 to 13.26, with 90% of the study sample having a score of 2.22 or less.) From survey data, we also assessed respondents’ educational attainment, general and mental health status, and whether a proxy responded on the beneficiary’s behalf.

STATISTICAL ANALYSIS

We used linear regression and a difference-indifferences approach to estimate changes in patients’ experiences in the ACO group from the preintervention period to the postintervention period that differed from concurrent changes in the control group and that were not explained by changes in observed sociodemographic and clinical characteristics of beneficiaries in the comparison groups. Specifically, for each item and composite measure, we fitted the following model:

• E(Score_i,t,k,h) = β₀ + β₁ACO_indicators_k + β₂Year_indicators_t + β_3kACO_Group × Postintervention_t + β₄HRR_indicators_h + β₅HRR_indicators_h × Year_indicators_t + β₆Covariates_i,

where E denotes the expected value, Score_i,t,k,h the score reported by patient i in survey year t assigned to ACO or non-ACO TIN k and residing in hospital referral region (HRR) h, ACO_indicators a vector of ACO indicators, Year_indicators a vector of survey-year indicators (omitting a reference year), ACO_Group assignment of the respondent to a provider group entering the Medicare ACO program in 2012, Postintervention the survey year 2013, HRR_indicators a vector of HRR indicators (omitting a reference region), and Covariates the sociodemographic and clinical characteristics in Table 2 (with age specified as a categorical variable and CCW conditions as 27 indicators).

Table 2.

Characteristics of Accountable Care Organization (ACO) and Control Groups before the Start of ACO Contracts in 2012.^*

Characteristic	ACO Group (N = 21,463)	Control Group (N = 186,846)
Age (yr)	74.8±8.6	74.8±8.6
Female sex (%)	54.1	53.3
Race or ethnic group (%)
White	85.9	86.2
Black	5.0	4.6
Hispanic	2.2	2.4
Other	6.9	6.8
Educational level (%)
Some high school or less	11.2	12.0
High-school diploma	34.7	35.2
Some college or 2-year college degree	28.0	27.2
4-year college degree	11.0	10.5
More than 4-year college degree	15.1	15.1
Disabled (%)†	8.4	8.6
End-stage renal disease (%)	0.4	0.3
CCW conditions‡
No. of conditions	5.37±3.39	5.39±3.41
≥6 conditions (%)	47.4	47.5
≥9 conditions (%)	18.1	18.3
HCC risk score§	1.09±0.94	1.08±0.93
Proxy survey respondent (%)	8.1	8.4
Self-reported health status¶
General health	3.05±0.98	3.04±0.99
Mental health	3.84±0.99	3.84±1.00

^*Plus–minus values are means ±SD. Means and percentages were adjusted for geographic region to reflect comparisons within hospital referral regions. Age, sex, race or ethnic group, disability status, presence or absence of end-stage renal disease, Chronic Conditions Data Warehouse (CCW) conditions, and Hierarchical Condition Category (HCC) scores were assessed from Medicare enrollment and claims data. Education, general health status, mental health status, and whether a proxy responded on a beneficiary’s behalf were assessed from CAHPS survey data. There were no significant between-group differences (P≥0.05) in the preintervention period, except for sex (P = 0.04) and for two categories of educational level (some high school or less [P = 0.007] and some college or 2-year college degree [P = 0.04]). The numbers of respondents shown indicate persons who responded to surveys conducted in the preintervention period of 2010 through 2012.

^†Data indicate the number of respondents for whom disability was the original reason for Medicare eligibility.

^‡Chronic conditions from the CCW include the following: acute myocardial infarction, Alzheimer’s disease, Alzheimer’s disease and related disorders or senile dementia, anemia, asthma, atrial fibrillation, benign prostatic hyperplasia, cataract, chronic kidney disease, chronic obstructive pulmonary disease, depression, diabetes, glaucoma, heart failure, hip or pelvic fracture, hyperlipidemia, hypertension, hypothyroidism, ischemic heart disease, osteoporosis, rheumatoid arthritis or osteoarthritis, stroke or transient ischemic attack, breast cancer, colorectal cancer, endometrial cancer, lung cancer, and prostate cancer.

^§HCC risk scores are derived from demographic and diagnostic data in Medicare enrollment and claims files, with higher scores indicating higher predicted Medicare spending. (In our study, HCC risk scores ranged from 0.12 to 13.26, with 90% of the study sample having a score of 2.22 or less.)

^¶Scores for self-reported health status range from 1 (poor) to 5 (excellent).

We did not include self-reported general and mental health in our main analyses because these survey measures could have been affected by ACO incentives. We included HRR and year indicators, and their interaction, to adjust for differences in the geographic distribution of the ACO and control groups and for region-specific changes in patients’ experiences. Thus, β₃ is the adjusted mean differential change in patients’ experiences reported in the ACO group as compared with local changes in the control group.

To facilitate interpretation, for each measure we calculated an effect size by dividing the differential change (β₃) by the standard deviation of mean scores across ACOs, using linear mixed models to estimate ACO-level standard deviations among patients that were in the ACO group (see the Supplementary Appendix). A differential improvement of 1 SD, for example, would be equivalent to moving from average performance (50th percentile) among ACOs to the 84th percentile. We also calculated effect sizes on the basis of standard deviations of mean scores across HRRs in the entire sample. Because the experiences of medically complex patients with high predicted health care utilization may be particularly affected by ACO efforts to improve quality and limit utilization,²³ in a prespecified subgroup analysis, we stratified the study sample into beneficiaries with seven or more CCW conditions and HCC risk scores of 1.10 or higher (approximately 25% of the beneficiaries) versus other beneficiaries (see the Supplementary Appendix). Because of differences in payment incentives, start dates, and organizational characteristics, we also estimated differential changes in patients’ experiences separately for beneficiaries assigned to Pioneer ACOs versus MSSP ACOs (see the Supplementary Appendix).

We conducted several analyses to address potential sources of bias. First, to adjust for any differences in trends evolving between the ACO group and the control group during the preintervention period, we added to models an interaction between the ACO group indicator and year, specified as a continuous predictor (i.e., we assumed trend differences would have continued into the postintervention period in the absence of ACO contracts). We present estimates both from models with adjustment and from models with no adjustment for differences in preintervention trends. Adjustment for preintervention trends could not be performed for most survey questions pertaining to care coordination and management because they were asked only in the 2012 and 2013 surveys (Table 1).

Second, we tested whether changes in the postintervention period for the control group were predicted by the preintervention trends of the group. Third, we compared sociodemographic and clinical characteristics between the ACO group and the control group before versus after the start of ACO contracts to gauge potential bias from differential changes in related characteristics that we could not measure. Fourth, as a falsification test, we excluded beneficiaries assigned to provider groups entering ACO contracts in 2012 and instead treated beneficiaries assigned to provider groups entering ACO contracts in 2013 as the ACO group; these groups were exposed to ACO incentives only for a few months (3.1 months on average) before the 2013 survey. In all analyses, we applied survey weights to adjust for survey nonresponse and oversampling of ACO patients in the 2013 survey and used design-based variance estimators to account for clustering within ACOs (in the ACO group) or HRRs (in the control group).²⁴

RESULTS

SURVEY RESPONDENTS

Among sampled beneficiaries with at least one primary care service, rates of response to the surveys conducted from 2010 through 2013 averaged 52.8%. Differences in response rates among beneficiaries assigned to ACOs versus among those assigned to other providers were small in both the preintervention period (+0.6 percentage points) and the postintervention period (−0.5 percentage points). Among the 32,334 respondents in the ACO group and the 251,593 respondents in the control group, observed sociodemographic and clinical characteristics in the preintervention period (Table 2) and any changes in these characteristics from the preintervention period to the postintervention period (Table S1 in the Supplementary Appendix) were similar in the two groups.

PATIENTS’ EXPERIENCES

During the preintervention period, patients’ ratings were similar in the ACO group and the control group for almost all measures (Table 3). For all the measures that had 3 years of preintervention data available, preintervention trends also were similar in the two groups (P≥0.17), and there were no significant postintervention changes in the control group after accounting for its preintervention trends (P≥0.17).

Table 3.

Differential Changes in Patients’ Experiences after the Start of ACO Contracts in 2012 for the ACO Group versus the Control Group.^*

Measure	Adjusted Means in Preintervention Period		Group Difference (95% CI)†	Adjusted Means in Postintervention Period		Group Difference (95% CI)	Differential Change in ACO Group		P Value	Differential Change Adjusted for Preintervention Trends‡		P Value
	ACO Group	Control Group		ACO Group	Control Group		Change (95% CI)	Effect Size§		Change (95% CI)	Effect Size§
Overall rating
Health care	8.59	8.59	0.00 (−0.03 to 0.03)	8.66	8.63	0.02 (−0.01 to 0.06)	0.02 (−0.02 to 0.06)	0.2	0.38	0.06 (−0.01 to 0.13)	0.5	0.10
Primary physician	9.04	9.04	0.00 (−0.03 to 0.02)	9.07	9.08	−0.01 (−0.05 to 0.04)	0.00 (−0.04 to 0.04)	0.0	0.99	0.02 (−0.05 to 0.09)	0.2	0.50
Specialist	8.94	8.93	0.01 (−0.02 to 0.04)	8.98	8.97	0.01 (−0.02 to 0.04)	0.01 (−0.03 to 0.05)	0.1	0.69	0.00 (−0.07 to 0.07)	0.0	0.95
Timely access to care	8.35	8.37	−0.02 (−0.05 to 0.01)	8.46	8.40	0.06 (0.02 to 0.11)	0.07 (0.02 to 0.13)	1.1	0.01	0.14 (0.02 to 0.25)	2.1	0.02
Interactions with primary physician	8.98	9.01	−0.03 (−0.06 to 0.00)	9.06	9.05	0.01 (−0.03 to 0.05)	0.03 (−0.01 to 0.08)	0.3	0.16	0.00 (−0.08 to 0.08)	0.0	0.93
Care coordination and care management
Primary physician informed about specialty care	7.82	7.87	−0.05 (−0.12 to 0.02)	7.94	7.85	0.10 (0.00 to 0.19)	0.14 (0.04 to 0.24)	0.5	0.006	0.29 (0.07 to 0.51)	1.0	0.01
Patient care information available to primary physician (1–10)	9.65	9.63	0.01 (−0.02 to 0.04)	9.64	9.63	0.01 (−0.02 to 0.04)	0.00 (−0.04 to 0.04)	0.0	0.96	NA	NA	NA
Communication of test results	8.70	8.71	−0.01 (−0.08 to0.07)	8.68	8.67	0.01 (−0.07 to 0.09)	0.02 (−0.06 to 0.10)	0.1	0.65	NA	NA	NA
Timely communication of test results	9.03	9.07	−0.04 (−0.10 to 0.02)	9.10	9.09	0.01 (−0.04 to 0.07)	0.05 (−0.01 to 0.12)	0.4	0.12	NA	NA	NA
Medication reconciliation	8.15	8.08	0.07 (−0.01 to 0.14)	8.12	8.10	0.01 (−0.06 to 0.08)	−0.06 (−0.15 to 0.04)	−0.3	0.25	NA	NA	NA
Patient access to visit notes — %	27.6	25.8	1.8 (0.1 to 3.5)	38.9	34.8	4.1 (2.1 to 6.1)	2.2 (0.6 to 3.9)	0.2	0.007	NA	NA	NA

^*Scores range from 1 to 10, with higher scores indicating better experiences with care. The measures of timely access to care and interactions with the primary physician are composite scores. Patients’ access to notes was assessed by means of a yes-or-no question; means are shown as percentages, with group differences and differential change shown as percentage points. Differential changes may not equal differences between the group differences because of rounding and slight differences in specification of models producing estimates of group differences and differential changes. CI denotes confidence interval, and NA not applicable.

^†All the between-group differences in the preintervention period were not significant (P>0.05), except for the differences in patients’ access to visit notes (P = 0.04).

^‡These differential changes were adjusted for any differences in trend between the ACO and control groups over the surveys from the period from 2010 through 2012. Because most questions about care coordination and management were asked only in the 2012 and 2013 surveys, no adjustment for prior trends could be made (i.e., not applicable).

^§Effect sizes were calculated by dividing the differential change by the standard deviation of ACO-level means. Thus, an effect size of 1 could be interpreted as moving from average performance among ACOs to approximately the 84th percentile among ACOs.

Overall ratings of care and physicians and ratings of interactions with primary physicians did not change differentially in the ACO group, as compared with the control group, from the preintervention period to the postintervention period (Table 3). In contrast, reports of timely access to care differentially improved in the ACO group (Table 3, and Fig. S1 in the Supplementary Appendix). These differential improvements were significant both when the analysis was adjusted and when it was not adjusted for group trends in the preintervention period (Table 3), with an effect size ranging from 1.1 SD (without adjustment for trends) to 2.1 SD (with adjustment for trends) of the ACO-level distribution, which corresponds to improvement from average performance among ACOs to the 86th to 98th percentile. Scores for two of six measures of care coordination and management (primary physician informed about specialty care and patient access to visit notes) also differentially improved in the ACO group (Table 3). Scores for the other four measures did not differentially change.

SUBGROUP ANALYSES

Overall ratings of care reported by patients in the ACO group with seven or more CCW conditions and HCC scores of 1.10 or higher improved significantly as compared with similarly complex patients in the control group (differential change, 0.11; 95% confidence interval [CI], 0.02 to 0.21; P = 0.02; differential change with adjustment for preceding trends, 0.20; 95% CI, 0.06 to 0.35; P = 0.005). These differential improvements in overall ratings of care corresponded to moving from average performance to the 82nd to 96th percentile among ACOs (effect size, 0.9 to 1.7 SD). In contrast, among patients considered to be less medically complex, those in the ACO group reported no significant differential change (Table S2 in the Supplementary Appendix). Differential improvements in other domains were not as clearly concentrated among more medically complex patients as differential improvements in overall ratings of care were (Table S2 in the Supplementary Appendix). Results were not significantly different for patients assigned to Pioneer ACOs versus those assigned to MSSP ACOs.

SENSITIVITY ANALYSES

Further adjustment for self-reported general and mental health status did not affect estimates appreciably. Effect sizes were similar when the calculation was based on standard deviations of mean scores across HRRs in the entire study sample. All significant differential improvements observed in our main analyses were smaller and not significant (P≥0.34) in sensitivity analyses (falsification tests) that replaced the ACO group with beneficiaries assigned to the 105 ACOs entering contracts in 2013.

DISCUSSION

In the first year of the Medicare ACO programs, incentives for participating provider organizations to limit health care utilization and improve quality of care were associated with meaningful improvements in some measures of patients’ experiences and with unchanged performance in others. As compared with local control groups of patients served by non-ACO providers, patients served by ACOs reported improvements in domains more easily affected by organizations (access to care and care coordination) but not in domains in which changes in physicians’ interpersonal skills may be required to achieve gains (interactions with physicians and physician ratings). In addition, medically complex patients, who were more likely to be the focus of ACO efforts to control utilization and enhance quality, reported significantly better overall care after the start of ACO contracts.

These findings have important implications for patients and policy. Enhanced experiences by patients may encourage their loyalty to ACOs, potentially addressing some of the care fragmentation and instability in beneficiary assignment that diminish incentives and rewards for ACOs.¹⁶ Moreover, should preliminary evidence of savings generated by ACOs be confirmed,^25–28 our findings would indicate that ACOs may be able to achieve savings in ways that do not adversely affect patients’ experiences. Finally, the improved experiences reported by patients in ACOs may constitute important initial progress by the Medicare ACO programs in fostering patient-centered, coordinated care. Accordingly, patients may benefit from choosing providers who are part of ACOs.

Our study has several limitations. In the first year, ACOs are required to report only on quality measures to maximize their shared savings rate (and minimize their shared loss rate). Thus, the effects of ACO contracts on patients’ experiences in subsequent years, when shared savings and losses depend on performance on quality measures,^2,3 may be greater than our estimates suggest. Because our study period included only the first 8 to 17 months of ACO contracts, we also had limited statistical power to assess differences between Pioneer ACOs and MSSP ACOs.

To facilitate interpretation, we calculated effect sizes in terms of standard deviations of ACO-level means. Minimal variation in patients’ experiences among ACOs could cause misleadingly large effect sizes, but variation among ACOs was similar to geographic variation in the study sample. In addition, the differential improvements in overall ratings of care reported by medically complex patients in our study were larger than previously reported differences in care ratings between Medicare fee-for-service and managed care programs.⁷ Moreover, variation among Medicare managed care plans in performance on CAHPS measures has been shown to predict much of the wide variation in plan-disenrollment rates.²⁹ Thus, although changes in mean scores regarding patients’ experiences may be difficult to interpret, available comparisons suggest that our estimates are meaningful.

Provider groups participating in the voluntary ACO programs differ in many respects from other providers. Some of our findings could be explained by differential responses of ACOs to other developments in health care markets that coincided with the start of the Medicare ACO programs (e.g., the Medicare and Medicaid Electronic Health Records Incentive Program).³⁰ Alternatively, ACOs already planning to implement systems to improve patients’ experiences may have been more likely to join the ACO programs than those without such plans. Enhancing access to care, for example, may yield financial gains for organizations under fee-for-service incentives. Nevertheless, patients’ experiences were improved or preserved in provider organizations participating in ACO programs despite incentives to limit health care use.

Moreover, for patients served by organizations entering the MSSP later (in 2013), we found no significant differential improvements, as compared with patients served by non-ACO providers, in access to care or in overall ratings of care among medically complex patients. Although this falsification test suggests that improvements achieved by ACOs entering the ACO programs in 2012 were related specifically to ACO contract incentives, the extent to which our findings are generalizable to the more than 200 ACOs entering the ACO programs after 2012 or to nonparticipating providers is unclear. Finally, the response rates to the CAHPS surveys averaged 52.8%, but differential changes in response rates and the characteristics of the respondents were minimal, suggesting that systematic bias from nonresponse was unlikely.

In conclusion, in the first year of the Medicare ACO programs, patients’ experiences did not deteriorate in any area assessed by the fee-for-service CAHPS survey, and they improved in areas that can be more readily modified by organizations and among patients likely to be targeted by ACO efforts to improve quality of care and to control utilization.