Analysis of Healthcare Cost and Utilization in the First Two Years of the Medicare Shared Savings Program Using Big Data from the CMS Enclave

Abstract

The Medicare Shared Savings Program (MSSP) is the larger of the first two Accountable Care Organization (ACO) programs by the Centers for Medicare and Medicaid Services (CMS). In this study we assessed healthcare cost and utilization of 1.71 million Medicare beneficiaries assigned to the 333 MSSP ACOs in the calendar years of 2013 and 2014, in comparison to years 2010 and 2011, using the official CMS data. We employed doubly robust estimation (propensity score weighting followed by generalized linear regression) to adjust the analyses to beneficiary personal traits, history of chronic conditions, previous healthcare utilization, ACO administrative region, and ZIP code socioeconomic factors. In comparison to the care delivered to the control cohort of 17.7 million non-ACO beneficiaries, we found that the care patterns for ACO beneficiaries shifted away from some costly types of care, but at the expense of increased utilization of other types, increased imaging and testing expenditures, and increased medication use, with overall net greater increase in cost instead of smaller increase.

Introduction

The Medicare Shared Savings Program (MSSP) was launched by the Centers for Medicare and Medicaid Services in 2012 as one of the first two Accountable Care Organization (ACO) programs as envisioned by the Patient Protection and Affordable Care Act (PPACA), which seek to coordinate and improve healthcare delivery by means of value- based reimbursement models. Considerable work has been done to study ACOs, including assessments of previously existing pilot programs and state-specific value-based reimbursement models^1–7, early studies of expected implications and qualitative analyses of impacts^8–10, studies focused on specific medical specialties^2,3,10–16, and more recent data- based reviews of the Pioneer ACO program, also launched in 2012^17,18.

In 2015, CMS made available, through their privacy-protecting Chronic Conditions Data Warehouse¹⁹ (CCW), the complete list of all beneficiaries assigned to and all providers enrolled in MSSP ACOs until December 31st, 2013. In 2016, the continuation of this dataset was released which extended data availability until the end of 2014. Combined, these two datasets cover all 333 ACOs that joined the MSSP between April 1st, 2012 and January 1st, 2014, and all Medicare beneficiaries assigned to them. The retrospective and detailed nature of the algorithm that CMS uses to assign beneficiaries to ACOs requires knowing the proportion of the total care received by each beneficiary that was delivered by ACO-enrolled providers, as well as the specialty of the physicians involved²⁰, and this information is difficult to infer without the authoritative datasets that CMS released in 2015 and 2016. As of July, 2016, there is a single publication³² analyzing the CMS MSSP data released in 2015, but no published analysis using both 2015 and 2016 datasets to assess the effect of the MSSP ACOs on resource utilization and cost. Here we report such an analysis. For simplicity, starting from this sentence the term “ACO” will always mean only MSSP ACO, never Pioneer ACO, except at the Discussion section of the paper.

Methods

Reproducible research

As a commitment to the transparency and reproducibility of our work, we voluntarily release for free download the complete source code used in this paper, under an Attribution-NonCommercial-ShareAlike 4.0 International²⁹ license, in an online GitHub repository at the link http://github.com/fabkury/cisa.It://github.com/fabkury/cisa.It is important to remind here that for reexecuting the code an interested researcher needs his/her own Data Use Agreement with CMS, by which one can receive access to the CMS claims files. The GitHub repository provides solely the source code and the full set of results produced by that source code.

Study design

We compared cost and utilization in the care provided to the beneficiaries assigned to ACOs with those not assigned using a difference-in-differences analysis of the trends in each group between before and after the start of the MSSP. The before period (baseline period) we defined as the combination of years 2010 and 2011, while the after period (outcome period) was the combination of 2013 and 2014. We ruled to completely ignore year 2012 as means of minimizing noise, upon recognizing that several of the most important variables we desired for analysis were available only at year-level granularity, while some ACOs had their agreement start date in April 1^st, 2012 and some in July 1^st, 2012. All ACOs from 2013 dataset (released in 2015) had their start date on or before January 1^st, 2013, and those on the 2014 dataset (released in 2016) had their start date on or before January 1^st, 2014. We additionally ruled to ignore any beneficiary that changed his or her U.S. state of residence at any moment between 2010 and 2014, as this could impart noise due to different healthcare pricing levels at different states.

Inpatient, Outpatient, Skilled Nursing Facility (SNF) and Physician Part B (Carrier) claims were aggregated by these four types as they are available in their separate CMS Research Identifiable Files²⁶ (RIFs). Components of care such as Intensive Care Unit days, hospice days, imaging and laboratory events and expenditures, Home Health visits, among others, were aggregated separately as available in the CCW Master Beneficiary Summary²¹ and MedPAR²² files, which identify these items using revenue codes.

Beneficiary eligibility for this study

To be sure that we had complete claims data needed for these analyses, we only included beneficiaries who were fully enrolled in Medicare Parts A and B during the entire years of 2010, 2011, 2013 and 2014, and had zero months of Health Maintenance Organization (HMO, Medicare Part C) enrollment in those four years. This also means that any beneficiary either not enrolled in Medicare since January 1^st, 2010, or not alive until December 31^st, 2014, was excluded. For defining the cohort of ACO-assigned beneficiaries, we further only considered those with ACO assignment during both complete calendar years of 2013 and 2014. For the cohort of non-ACO beneficiaries, we considered only those never assigned to an ACO at any moment.

Statistical analysis

We first used the propensity score approach to mitigate differences in beneficiary-level characteristics between those enrolled in ACO and those not. We employed a binary logistic regression model to calculate the predicted probability of enrolling in an ACO using 66 covariates obtained from the baseline period (years 2010 and 2011). Covariates used in the propensity score included beneficiary-level characteristics such as age (at the end of 2011), gender, race, presence of 26 CMS predefined priority chronic conditions²¹ (by the end of 2011), region of residence as defined by the ACO regional offices²³, reason for Medicare eligibility, and several separate counts and costs of drug and nondrug care received. Each possible region of residence, chronic health condition, race, reason for Medicare eligibility, as well as gender, was a binary flag. The remaining covariates were numerical values.

We then used a generalized linear regression model with the propensity score weight²⁴ to compare outcomes of interest between those enrolled in ACOs and those not. The propensity score weight was defined as the inverse of the predicted probability of enrolling in an ACO. Individuals with a high predicted probability of enrolling in ACOs, who actually enrolled in ACOs, receive a lower weight compared with individuals with a low probability of enrolling in ACOs. In this regression model, outcomes of interest from the ACO and the non-ACO groups were compared not only weighted by the propensity score, but also controlled again for beneficiary-level and also beneficiary ZIP code-level characteristics, forming a doubly robust estimator³³. The generalized linear regression model included 48 covariates, of which 45 were also included in the propensity score. In total, the statistical analyses adjusted for 69 covariates. We conducted all statistical analyses using SAS software, version 9.4 (SAS Institute Inc., Cary, NC), inside the CMS Virtual Research Data Center²⁵ (CMS “Enclave”).

For economy of space, the covariates included in the propensity score and in the linear regression model are specified together with their balances before and after the propensity score weighting in Table 1 at the Results section. All 26 chronic health conditions²¹ were included in both propensity score and linear regression, but were omitted from the table in this paper for economy of space – they are available in the full set of results for download at the online repository.

Table 1.

Cohort characteristics and covariates used in the statistical analyses, along with average values pre and post propensity score weighting, except for the 26†‡ chronic health conditions²¹. Numbers are average per beneficiary in years 2010 and 2011 ± standard error. The binary flags are indicated by the absolute count followed by the percentage of the total that the count represents.

	No PS weighting		With PS weighting
Variable	Non-ACO	ACO	Non-ACO	ACO
†‡Beneficiary age at the end of 2011	70.859 ± 12.534	72.723 ± 10.817	71.022 ± 13.015	71.057 ± 39.851
†‡Female gender	9,604,042 (54.27%)	998,475 (58.51%)	9,671,501 (54.65%)	954,972 (55.96%)
†‡White	14,969,286 (84.59%)	1,499,954 (87.90%)	15,021,505 (84.88%)	1,455,965 (85.32%)
†‡Black §	1,693,781 (9.57%)	126,223 (7.40%)	1,660,070 (9.38%)	160,380 (9.40%)
†‡Hispanic §	340,354 (1.92%)	20,635 (1.21%)	329,642 (1.86%)	31,978 (1.87%)
†‡Asian	275,298 (1.56%)	32,341 (1.90%)	279,704 (1.58%)	23,114 (1.36%)
Other ethnicity	418,412 (2.36%)	27,269 (1.60%)	406,210 (2.30%)	34,986 (2.05%)
†‡ACO region 1: MA, VT, NH, ME, RI, CT	924,786 (5.23%)	220,998 (12.95%)	1,046,749 (5.92%)	103,225 (6.05%)
†‡ACO region 2: NY, NJ, PR, USVI	1,617,822 (9.14%)	229,682 (13.46%)	1,685,348 (9.52%)	157,533 (9.23%)
†‡ACO region 3: PA, DE, WV, VA, DC, MD	1,922,113 (10.86%)	117,078 (6.86%)	1,859,328 (10.51%)	180,397 (10.57%)
†‡ACO region 4: GA, KY,TN,NC,SC, AL, MS, FL §	3,885,810 (21.96%)	365,902 (21.44%)	3,877,755 (21.91%)	373,615 (21.90%)
†‡ACO region 5: IL, MN, WI, MI, IN, OH	2,859,615 (16.16%)	393,841 (23.08%)	2,967,123 (16.77%)	288,871 (16.93%)
†‡ACO region 6: TX, NM, OK, AR, LA §	2,189,383 (12.37%)	88,179 (5.17%)	2,076,514 (11.73%)	200,563 (11.75%)
†‡ACO region 7: MO, NE, IA, KS	960,840 (5.43%)	127,170 (7.45%)	992,413 (5.61%)	98,240 (5.76%)
†‡ACO region 8: CO, MT, ND, WY, SD, UT §	575,102 (3.25%)	19,051 (1.12%)	541,703 (3.06%)	52,351 (3.07%)
†‡ACO region 9: CA, NV, AZ, HI, GU, AS, MP	2,020,863 (11.42%)	106,485 (6.24%)	1,940,036 (10.96%)	183,422 (10.75%)
†‡ACO region 10: WA, OR, ID, AK §	740,797 (4.17%)	38,036 (2.23%)	710,162 (4.01%)	68,206 (4.00%)
†‡Medicare-elig. due to age >= 65	14,063,975 (79.47%)	1,461,251 (85.63%)	14,159,233 (80.01%)	1,368,480 (80.20%)
†‡Medicare-elig. due to disability	3,545,624 (20.04%)	238,818 (14.00%)	3,450,985 (19.50%)	328,552 (19.25%)
†‡Medicare-elig. due to end-stage renal disease	39,273 (0.22%)	2,479 (0.16%)	39,298 (0.22%)	4,069 (0.24%)
Medicare-elig. due to disab. and renal disease	48,259 (0.27%)	3,874 (0.23%)	47,616 (0.27%)	5,322 (0.31%)
‡ZIP code pop. below Federal Poverty Level	0.154 ± 0.095	0.133 ± 0.089	0.153 ± 0.099	0.141 ± 0.303
‡ZIP code pop. >=25 years with High School degree	0.862 ± 0.089	0.88 ± 0.081	0.863 ± 0.092	0.871 ± 0.298
‡ZIP code pop. >=25 years with College degree	0.274 ± 0.162	0.306 ± 0.16	0.276 ± 0.17	0.298 ± 0.545
Number of chronic conditions at end of 2011	5.475 ± 3.752	6.581 ± 3.293	5.575 ± 3.925	5.88 ± 11.441
Months of Part D Coverage	11.524 ± 11.758	12.334 ± 11.762	11.577 ± 12.318	12.42 ± 39.167
†‡ Part D Fill Count a	47.822 ± 73.194	55.033 ± 75.034	48.492 ± 77.121	51.977 ± 243.641
†‡ Part B Physician Events	13.132 ± 14.271	17.525 ± 14.611	13.569 ± 15.9	14.916 ± 42.823
†‡ Part B Drug Events	4.666 ± 9.974	5.825 ± 10.314	4.79 ± 12.068	5.212 ± 34.986
†‡ Ambulatory Surgery Center Events	0.415 ± 1.483	0.536 ± 1.591	0.432 ± 2.188	0.461 ± 4.64
(Evaluation and Management Events	8.044 ± 19.244	9.692 ± 20.129	8.23 ± 21.224	9.147 ± 70.693
†‡ Anesthesia Events	0.695 ± 1.555	0.871 ± 1.731	0.711 ± 1.64	0.775 ± 8.437
†Dialysis Events §	0.146 ± 2.177	0.135 ± 2.05	0.186 ± 5.566	0.183 ± 7.583
†Other Procedures Events	10.048 ± 25.26	13.464 ± 29.623	10.383 ± 27.361	11.396 ± 88.324
†Imaging Events	6.743 ± 8.905	8.472 ± 9.167	6.904 ± 9.441	7.5 ± 29.456
†Tests Events	23.494 ± 33.387	31.207 ± 35.044	24.264 ± 37.27	26.695 ± 103.18
†Durable Medical Equipment Claims	4.312 ± 11.649	4.948 ± 11.986	4.373 ± 12.251	4.708 ± 40.326
†Other Part B Carrier Events	2.544 ± 13.463	3.158 ± 13.422	2.606 ± 14.313	2.797 ± 43.545
†Hospital Outpatient Visits	11.051 ± 29.484	12.293 ± 27.996	11.189 ± 30.664	12.547 ± 119.768
†Home Health Visits	3.935 ± 27.295	3.892 ± 25.294	3.936 ± 28.215	4.474 ± 113.523
†Emergency Room Visits (inpatient or outpatient)	0.902 ± 2.52	0.952 ± 2.317	0.907 ± 2.608	1.021 ± 13.19
†Hospice Covered Days	0.355 ± 11.637	0.237 ± 9.274	0.344 ± 11.974	0.356 ± 39.028
†Skilled Nursing Facility Covered Days	1.651 ± 10.871	1.854 ± 11.118	1.67 ± 11.408	1.802 ± 38.348
†Inpatient Covered Days	2.05 ± 7.816	2.179 ± 7.518	2.068 ± 8.166	2.279 ± 30.371
†30-day Hospital Readmissions	0.047 ± 0.374	0.053 ± 0.381	0.048 ± 0.392	0.057 ± 1.783
†Total Beneficiary Payments	2,765.639 ± 4,389.09	3,302.88 ± 4107.52	2,819.21 ± 4,633.21	3,074.881 ± 14,701.60
†Total Medicare Payments	16,974.18 ± 32,809.41	19,579.81 ± 31,482.99	17,245.46 ± 34,514.71	18,912.58 ± 111,148.80

^†Variable included as covariate in propensity score.

^‡Variable included as covariate in generalized linear regression model.

^§:P-value for the comparison post-propensity score weighting is above 0.05. All others are under 0.05.

^¤Defined by the CMS as the number of 30 days of individual drug fills. 12 months of one drug produce fill count of 12. 12 months of 2 drugs produce 24. For more information, please see the official definition at the ResDAC website²¹.

Outcome measures

We considered an extensive list of 151 outcome measures, as well as separate measurements of number of claims and total cost of filled drug prescriptions aggregated by 493 different ATC-4 classes that matched the NDCs present in the Medicare Part D claims of the beneficiaries included in the study, totaling 1,137 comparisons between the ACO cohort and the Non-ACO cohort. Due to the obvious lack of space, we opted to include and discuss in this paper only the differences that were most insightful to our opinion. The selected measures are depicted in Tables 2, 3 and 4 at the Results section. The complete list of outcome measures can be downloaded, along with our complete source code, at the online repository.

Table 2.

Claims aggregated by type of healthcare facility. Numbers are differences in average per beneficiary between the baseline and outcome periods

Measure	Non-ACO	ACO	Difference	Percent diff.
Carrier (physician Part B) claims	5.544 ± 0.0129	9.385 ± 0.0133	-3.8406	-69.27%
Carrier (physician Part B) total charges ($)	679.30 ± 2.22	1033.57 ± 2.28	-354.27	-52.15%
Outpatient claims	1.379 ± 0.0033	1.534 ± 0.0034	-0.1545	-11.20%
Outpatient total charges ($)	10,077.73 ± 28.41	11,036.73 ± 29.22	-959	-9.52%
Inpatient total charges ($)	6,270.27 ± 26.3594	5,610.71 ± 27.12	659.55	10.52%
Inpatient claims	0.08 ± 0.0004	0.071 ± 0.0004	0.0089	11.10%
Skilled Nursing Facility total charges ($)	1,287.48 ± 4.28	886.70 ± 4.41	400.79	31.13%
Skilled Nursing Facility claims	0.105 ± 0.0004	0.068 ± 0.0004	0.0366	34.86%

Table 3.

Components of care at inpatient and skilled nursing facility settings. Numbers are differences in average per beneficiary between the baseline and outcome periods.

Measure	Non-ACO	ACO	Difference	Percent diff.
Coronary care unit, pulmonary claims †	0.000037 ± 0.000006	0.000056 ± 0.000006	-0.0000195	-52.53%
Coronary care unit, intermediate claims	0.006172 ± 0.000098	0.008709 ± 0.0001	-0.0025371	-41.11%
ICU, intermediate IOU claims	0.017504 ± 0.000138	0.022418 ± 0.000142	-0.0049134	-28.07%
Radiology therapeutic claims	0.000498 ± 0.00002	0.000622 ± 0.00002	-0.0001236	-24.80%
Coronary Care Unit days ‡	0.037944 ± 0.000546	0.039248 ± 0.000562	-0.0013036	-3.44%
MRI charges ($)	58.94 ± 0.60	57.43 ± 0.62	1.5083662	2.56%
Surgical procedures §	0.030685 ± 0.000299	0.029715 ± 0.000307	0.0009700	3.16%
Laboratory charges ($)	766.73 ± 3.16	682.06 ± 3.25	84.67	11.04%
Total charges ($)	7,281.28 ± 27.85	6,382.92 ± 28.66	898.36	12.34%
Coronary care charges ($)	196.04 ± 2.12	171.73 ± 2.18	24.31	12.40%
Radiology charges ($)	436.42 ± 1.90	379.79 ± 1.95	56.63	12.98%
Pharmacy charges ($)	710.15 ± 4.87	615.54 ± 5.01	94.61	13.32%
ICU, general claims	0.00879 ± 0.0001	0.007477 ± 0.000103	0.0013127	14.93%
Radiology other imaging claims	0.015039 ± 0.00013	0.012313 ± 0.000134	0.0027263	18.13%
General drugs and/or IV therapy claims	0.109558 ± 0.000506	0.089017 ± 0.000521	0.0205408	18.75%
ICU, medical claims	0.001252 ± 0.000035	0.000889 ± 0.000036	0.0003639	29.06%
ICU, surgical claims	0.000636 ± 0.000031	0.000416 ± 0.000032	0.0002201	34.59%
Informational encounters	0.000054 ± 0.000003	0.000029 ± 0.000003	0.0000252	46.49%
Radiology nuclear medicine claims	0.00141 ± 0.000082	0.000581 ± 0.000085	0.0008288	58.79%
ICU, psychiatric claims	-0.000193 ± 0.000017	-0.000322 ± 0.000017	0.0001288	66.63%
Ambulatory surgical care claims	-0.000245 ± 0.000022	-0.000412 ± 0.000023	0.0001672	68.15%
ICU, other claims	0.000269 ± 0.000022	0.000062 ± 0.000023	0.0002071	76.84%
Coronary care unit, heart transplant claims	0.000013 ± 0.000002	-0.000004 ± 0.000002	0.0000165	127.82%
ICU, pediatric claims	0.000018 ± 0.000002	-0.000013 ± 0.000002	0.0000315	174.51%
Coronary care unit, other claims ¤	-0.00002 ± 0.000018	-0.000059 ± 0.000018	0.0000387	189.87%
Coronary care unit, myocardial claims	-0.000007 ± 0.000006	-0.000031 ± 0.000006	0.0000242	351.30%

^† P-value of 0.029. ‡: P-value of 0.096. §: P-value of 0.024. ¤: P-value of 0.126.

Table 4.

Components of care and total costs across all care settings. Numbers are differences in average per beneficiary between the baseline and outcome periods.

Measure	Non-ACO	ACO	Difference	Percent diff.
Ambulatory Surgery Center Medicare Payments ($)	7.2 ± 0.27	36.17 ± 0.28	-28.97	-402.08%
Durable Medical Equipment Medicare Payments ($)	-8.14 ± 0.46	6.45 ± 0.47	-14.6	-179.25%
Other Procedures Medicare Payments ($)	57.98 ± 0.83	151.64 ± 0.85	-93.66	-161.55%
Part B Physician Events	1.038 ± 0.004	2.643 ± 0.004	-1.605	-154.54%
Tests Medicare Payments ($)	33.5 ± 0.27	78.61 ± 0.28	-45.11	-134.66%
Dialysis Events	0.03 ± 0.001	0.067 ± 0.001	-0.037	-123.65%
Ambulatory Surgery Center Events	-0.061 ± 0.001	0.014 ± 0.001	-0.075	-122.18%
Imaging Medicare Payments ($)	-28.84 ± 0.2	2.99 ± 0.21	-31.83	-110.35%
Other Procedures Events	0.795 ± 0.009	1.619 ± 0.009	-0.824	-103.73%
Tests Events	2.486 ± 0.01	4.994 ± 0.01	-2.509	-100.93%
Part B Drug Events	0.48 ± 0.004	0.914 ± 0.004	-0.434	-90.52%
Part B Physician Medicare Payments ($)	113.21 ± 0.23	207.38 ± 0.23	-94.17	-83.18%
Imaging Events	0.517 ± 0.003	0.928 ± 0.003	-0.411	-79.49%
Anesthesia Medicare Payments ($)	11.06 ± 0.09	19.46 ± 0.09	-8.39	-75.87%
Anesthesia Events	0.077 ± 0.001	0.134 ± 0.001	-0.057	-73.37%
Durable Medical Equipment Claims	0.367 ± 0.003	0.614 ± 0.003	-0.247	-67.28%
Part B Drug Medicare Payments ($)	213.48 ± 2.11	277.51 ± 2.17	-64.02	-29.99%
Part D Medicare Payments ($)	871.95 ± 3.56	1051.71 ± 3.66	-179.76	-20.62%
Part D Total Prescription Costs ($)	1248.55 ± 3.82	1486.8 ± 3.93	-238.25	-19.08%
Part D Fill Count f	20.732 ± 0.017	23.903 ± 0.017	-3.171	-15.29%
Total Beneficiary Payments ($)	805.88 ± 1.58	889.37 ± 1.62	-83.49	-10.36%
Hospital Outpatient Medicare Payments ($)	815.16 ± 2.7	897.53 ± 2.77	-82.37	-10.10%
Total Medicare Payments ($)	5,950.14 ± 12.35	6,340.91 ± 12.71	-390.77	-6.57%
Evaluation & Management Medicare Payments ($)	223.81 ± 0.56	238.32 ± 0.58	-14.52	-6.49%
Dialysis Medicare Payments ($)	9.1 ± 0.09	8.43 ± 0.09	0.67	7.32%
Other Part B Carrier Medicare Payments ($)	101.44 ± 0.64	93.83 ± 0.65	7.61	7.50%
Emergency Room Visits (inpatient or outpatient)	0.237 ± 0.001	0.217 ± 0.001	0.021	8.74%
Hospital Outpatient Visits	2.664 ± 0.009	2.419 ± 0.009	0.245	9.18%
Home Health Medicare Payments ($)	300.466 ± 1.107	261.3 ± 1.14	39.17	13.04%
Home Health Visits	1.874 ± 0.009	1.529 ± 0.009	0.345	18.42%
Hospice Medicare Payments ($)	240.12 ± 1.21	153.48 ± 1.24	86.64	36.08%
Skilled Nursing Facility Covered Days	1.408 ± 0.005	0.89 ± 0.005	0.518	36.80%
30-day Hospital Readmissions	0.016 ± 0.0002	0.01 ± 0.0002	0.006	36.99%
Hospice Covered Days	1.58 ± 0.008	0.986 ± 0.008	0.595	37.63%
Skilled Nursing Facility Medicare Payments ($)	577.46 ± 2.23	357.97 ± 2.29	219.49	38.01%
Inpatient Covered Days	0.567 ± 0.003	0.347 ± 0.003	0.22	38.84%
Other Inpatient Medicare Payments ($)	196.04 ± 2.1	77.65 ± 2.16	118.4	60.39%

^†Defined by the CMS as the number of 30 days of individual drug fills. 12 months of one drug produce fill count of 12. 12 months of 2 drugs produce 24. For more information, see the ResDAC website²¹.

Part D drug claims and cost measures

The measurements of drug claims and costs aggregated by their ATC-4 classes (which were mapped from the drugs’ NDC codes) are not presented in this paper. They are available only for download at the online repository. We made this decision for three reasons. First, we preferred to prioritize other content in our limited space in this paper. Second, this specific part of the analysis relies solely on a 10% purely random sample of the beneficiaries, which is the sub-cohort from whom we have Part D claims as per our Data Use Agreement with CMS, while all other parts of this paper enjoy access to the totality of Medicare beneficiaries that met our eligibility criteria for this study. Third, the costs of Part D claims do not count towards the performance of a MSSP ACO as per the program’s definition, therefore the ACOs have no direct incentive to save on expenses that are billed to Part D. None of these three reasons are the case of drugs administrated inside health care facilities and billed to Medicare Part B. The costs of all drugs billed to Part B for all beneficiaries are included in the drug usage measurements in Table 3. Table 4 contains, separately, the total sum of costs of drugs billed to both Medicare Parts B and D and covers the totality of beneficiaries (not 10%).

Exclusion of outlier claims

Because our analyses processed hundreds of millions of individual claims from millions of beneficiaries, we were concerned about potential outlier values among the claims, e.g. data entry errors or faulty data for any reason. In order to mitigate the potential impact of unknown outliers on our calculations of averages, we implemented a mechanism for ignoring extreme outlier claims. Any claim whose cost ranked at the top 0.01^th percentile (0.01%, not to be confused with top 1^st percentile) of its data source (i.e. each CMS file available in the VRDC) was deemed a potential outlier and therefore ignored.

Data Sources

This study was performed using the CMS claims data available to our group inside the CMS Virtual Research Data Center²⁵ via our Data Use Agreement #26250. The claims files used were full-size (100% of the Medicare beneficiaries) Inpatient RIF, Outpatient RIF, Skilled Nursing Facility RIF, Carrier RIF, MedPAR, and Master Beneficiary Summary File segments A/B, Chronic Conditions, and Cost & Use²⁶. Socioeconomic variables for each ZIP Code Tabulation Area (ZCTA) were retrieved from the FactFinder tool at Census.gov²⁷. The mapping of beneficiary ZIP code to ZCTA was performed using the crosswalk table produced by the Robert Graham Center upon funding by the Health Resources and Services Administration²⁸. The mapping of Medicare Part D drugs from their NDC codes to ATC-4 classes was performed by querying the RxNorm API web service³⁴ using an R language script, which is also available for download in the online repository.

Results

After applying all restraints described in the Methods section, we had a cohort of 1,708,888 ACO-assigned beneficiaries and a control cohort of 17,741,210 Non-ACO beneficiaries. The outcomes included 151 measurements plus 986 drug class utilization and cost measurements, so the total number of outcome comparisons between the ACO cohort and the non-ACO cohort was 1,137. Due to the very large size of the studied cohorts, the p-values were very small, although in many cases the identified difference was also small if not frankly negligible. In the great majority of the cases, the resulting p-value was smaller than 10^-6, so we thought there was little need to perform corrections for multiple-testing, as it was unlikely that we would arrive at a non-negligible probability of having committed a statistical Type I (“false positive”) error. Unless otherwise indicated, a p-value below 10^-6 is to be assumed for all comparisons included in this paper, except for those in Table 1. The p-values of all comparisons (including those of Table 1) can be found, rounded to 9 decimal places, in complete set of results available for download at the online repository. All monetary values are expressed in U.S. dollars.

Covariate balancing by the propensity score

Table 1 depicts all variables included in the propensity score (PS) - except for the 26 CCW chronic conditions²¹, here omitted for economy of space, which were also included - and their balance before and after weighting. Due to the very large number of beneficiaries in both ACO and non-ACO cohorts, even despite the small differences in the averaged values after weighting, all but 9 p-values when comparing the two groups post-weighting were below 10^-6. Out of the 69 covariates considered, 58 had a p-value below 10^-6 even after the propensity score weighting. Despite the low p-values, as one could expect from an analysis totaling 19.45 million individuals, the propensity score was effective in balancing their means between the ACO and non-ACO cohorts. The full list of covariates, including the 26 chronic health conditions and all p-values, can be downloaded at the GitHub repository.

Comparison of overall claims by type of healthcare facility

Table 2 shows that the average number of claims per beneficiary, as well as average cost per beneficiary, increased more in the ACO cohort than in the non-ACO in outpatient and physician Part B (carrier) settings, while at inpatient and skilled nursing facility settings it was the opposite. The numbers under the Non-ACO and ACO columns in Tables 2, 3 and 4 are the difference between the average per beneficiary in the baseline and outcome periods, i.e.average per beneficiary at outcome periodminusaverage per beneficiary at baseline period, along with the standard error of the average. The Difference column is defined as Non-ACO value minus ACO value, while the Percent difference column is [Non-ACO value minus ACO value] divided by unsigned Non-ACO value. Therefore, in those three tables, negative differences (as well as negative percent differences) indicate that the average per beneficiary increased more in the ACOs cohort than in the non-ACO cohort from the baseline (before) period to the after (outcome) period.

omparison of components of care at inpatient and skilled nursing facility settings

Table 3 depicts some of the outcome measures available from the MedPAR file, which covers inpatient and skilled nursing facility (SNF) claims. It seemingly shows a trend of health care shifting away from expensive types of care such as stays in heart transplant and myocardial infarction specialized Coronary Care Units (CCUs), at the expense of increased use of intermediate CCUs and ICUs and increased expenditures with therapeutic Radiology. In these two care settings (inpatient and SNF), diagnostic radiology claims, laboratory charges and ambulatory surgical care claims increased faster in the Non-ACO cohort. The overall net effect here was positive for the ACO cohort, i.e. smaller increase in total cost.

Comparison of components of care in all care settings as available in the Master Beneficiary Summary File (MBSF)

The Master Beneficiary Summary File (MBSF) has an important advantage over all other files as it provides data already aggregated, per beneficiary, from the entirety of the CMS claims regardless of whether your specific Data Use Agreement with CMS covers the files from which the claims were retrieved. For example, our DUA does not cover Durable Medical Equipment claims, notwithstanding their aggregated events and costs are available, albeit at year- level granularity per beneficiary, in the MBSF. This also means that the MBSF provides the most reliable measure of total costs, which, as depicted in Table 4, we found to have increased faster in the ACO cohort than in the non-ACO cohort ($390.77 greater increase in average per beneficiary). In some cases, the non-ACO cohort saved money in comparison to its own baseline, while the ACO cohort did not.

Discussion

For the most part our data suggests that the cost for the care given to beneficiaries assigned to MSSP ACOs increased faster from the baseline period (2010 plus 2011) to the outcomes period (2013 plus 2014) than the cost of a control cohort, even after adjusting for a total of 69 covariates in a doubly robust estimator – propensity score weighting followed by re-adjustment in a generalized linear regression model. The covariates included spanned beneficiary personal traits, medical history, ACO administrative region, and socioeconomic factors of the beneficiary’s ZIP code of residence, in correspondence to the covariates used in previous studies^17,32.

It is expected to see generalized increases in average number of claims per patient, since the fixed cohorts will have aged from 2010 to 2014; and it is expected to find generalized increase in costs since, on top of the increased number of claims, the monetary values are not adjusted for inflation. It is hard to explain, though, why the ACO cohort exhibited net greater increase in total cost than the non-ACO cohort.

Our results are different than those from studies of the Pioneer ACO program^17,18 and the Blue Cross Blue Shield of Massachusetts’ Alternative Quality Contract^1,2,4,6,7, and the recent analysis of the first year (2013) of the Medicare Shared Savings Program³². Our results suggest an opposite effect which we cannot easily explain. McWilliams et al. ³² identified savings among the MSSP ACOs with agreement start date earlier than January 1^st, 2013, but not among the ones that started precisely on January 1^st, 2013, suggesting that the capacity to generate savings might take some time to be developed. In our study, however, after ignoring year 2012 and analyzing the claims of all 333 MSSP ACOs up until the end of 2014, we found negative savings. Moreover, it may also be worthy of notice that McWilliams et al. emulated as possible the CMS ‘ algorithm for retrospectively assigning beneficiaries to MSSP ACOs using the official list of MSSP ACO providers, and did so using a 20% random sample. In our study we did not consider the list of MSSP ACO providers. We used only the official list of beneficiaries assigned to MSSP ACOs as provided by CMS via the CMS Virtual Research Data Center (VRDC), and had access to the totality of beneficiaries, albeit our study design drastically reduced the number of beneficiaries considered due to our eligibility criteria.

In an effort to assure complete data for all of our outcomes of interest, we excluded beneficiaries with anything less than complete data in all years under study, as described in the subsection Beneficiary eligibility for this study. The eligibility criteria resulted in the loss of 46.5% of the beneficiaries enrolled in Medicare in all four years considered, and inclusion of only 36.5% of the beneficiaries enrolled in Medicare during the study period that were ever assigned to a MSSP ACO, as depicted in Figure 1. In future analyses we plan to look for unbalances in the patients who were excluded from this study that might help explain our present results.

Figure 1.

Beneficiary eligibility

The figures presented by The Centers for Medicare and Medicaid Services for MSSP SCOs were positive^30,31 in comparison to the benchmark defined for each MSSP ACO, with $705 million saved by 58 MSSP ACOs in Performance Year (PY) 2013 and $806 million saved in PY 2014 by 92 MSSP ACOs. However, the methods used by the MSSP for quantifying savings are quite different from those of this paper. The MSSP compares actual post-ACO expenditures against ACO-specific projected expenditures²⁰ derived from the historical, fee-for-service Medicare Parts A and B expenditures, specific to the beneficiaries assigned to each ACO, while adjusting for beneficiary characteristics and other factors and correcting for the projected growth in national per capita health expenditures. The details of these projection methods can be found in the references²⁰.

Conclusion

Our results suggest that over its first two years the MSSP ACOs may have increased the costs of Medicare spending when comparing to an adjusted control cohort of beneficiaries never assigned to a MSSP ACO, but these results contradict other studies on other types of value-based healthcare reimbursement models, and also a recent study of the first year of the MSSP. Further analysis is needed to understand the differences between our results and previous studies.

Disclaimer

The views and opinions of the authors expressed herein do not necessarily state or reflect those of the U.S. National Library of Medicine, National Institutes of health or the U.S. Department of Health and Human Services. This research was supported by the Intramural Research Program of the National Institutes of Health (NIH), National Library of Medicine (NLM), and Lister Hill National Center for Biomedical Communications (LHNCBC). This research was also supported in part by an appointment to the NLM Research Participation Program, administered by the Oak Ridge Institute for Science and Education (ORISE) through an interagency agreement between the U.S. Department of Energy (DoE) and the NLM.