Cost of Practice Transformation in Primary Care: Joining an Accountable Care Organization

Abstract

The purpose of this study is to examine the costs related to practice transformation from the perspective of primary care organizations transitioning to become participants in Accountable Care Organizations (ACOs). We pose two research questions: 1) Will a Rural Health Clinic that participates in an Accountable Care Organization see higher or lower cost per visit, and 2) If the cost per visit is higher or lower, how large will that difference be?

We analyze administrative data from a panel of over 800 Rural Health Clinics for the period 2007 – 2013 using a treatment effects approach, where a clinic’s participation in an ACO is viewed as a “treatment.” Since the first year that an RHC could join an ACO was 2012 and our most recent year of complete data is 2013, we restricted our analysis of the impact of participation in an ACO to include only 2012 and 2013 data. The estimates of the average treatment effect on the treated (ATET) pertain to only those RHCs that joined ACOs. The results show that those 20 sample ACO RHCs experienced an average from $15.00 to $18.61 higher cost per visit than the matching non-ACO RHCs.

At this very early stage of ACO development, our results must be considered very preliminary at best. Whatever conclusions we draw from these results are intended to merely suggest what might be found once many more RHCs join ACOs. The conclusions we draw from this early analysis can lay a foundation for more analysis after data are available when more RHCs join ACOs.

Introduction

Determining how primary care services can be best organized has been identified as a research need by rural stakeholders.¹ This need has developed partly in response to several fundamental changes in the healthcare environment that are affecting rural and urban healthcare providers today. Among these are a shift in payment policies from a focus on the volume of services to a focus on the quality of those services; changes in quality measures used by payers such as Medicare, Medicaid, and private insurers; and the growth of competing healthcare provider sectors such as retail clinics.

Accompanying these changes is a trend toward mergers and consolidations among healthcare organizations as they anticipate growth of Accountable Care Organizations (ACOs) throughout the country.² ACOs are groups of healthcare providers, such as combinations of hospitals, primary care organizations, and specialists, or groups of primary care organizations, that come together in a formal arrangement to provide high quality care to a defined population of patients while reducing the per capita costs of that care. Primary care providers are viewed as essential in this new model for healthcare delivery.³

In rural areas, Rural Health Clinics (RHCs) are increasingly affected by those fundamental changes. RHCs are primary care clinics certified by means of the Rural Health Clinic Program, which was established in 1977 to improve access to primary care in underserved rural areas. As RHCs and other primary care providers plan for the organizational transformation of joining ACOs or other multiple-provider organizations, they will need to anticipate the costs of that transformation, such as the cost of implementing electronic health record systems, implementing new quality measures, and training staff. To date there is scarce empirical evidence of the effects of practice transformation on the costs of primary care organizations.

The purpose of this study is to examine the costs related to practice transformation from the perspective of primary care organizations transitioning to become ACO participants. We pose the following research questions:

1. Will a Rural Health Clinic that participates in an Accountable Care Organization see higher or lower cost per visit?

2. If the cost per visit is higher or lower, how large will that difference be?

Practice Transformation

The term “practice transformation” is often used to describe the process of changing patient care programs, or implementing new ways of organizing care. In this study we use a broader interpretation of the term, where “practice transformation” (also called “primary care redesign” or “practice change”) describes change within a complex adaptive system.⁴ We apply the term at the organizational level to describe the experiences of primary care organizations as they transition to become members of a multiple-provider healthcare system – the ACO. We analyze the differences in costs for a sample of primary care organizations that chose to participate in ACOs versus those of a comparable group that did not participate in ACOs.

To date, few studies analyze primary care organizations’ transformation to become participants of groups of health care providers such as ACOs. Fewer still analyze the costs associated with those transformation processes from the primary care organization perspective. Studies on costs experienced by healthcare organizations that have joined or are considering joining ACOs are often descriptive or advisory in nature. Pettin⁵, for example, stresses the importance of ACO participants monitoring their operating and care delivery costs, as well as the relationship between those costs and patient outcomes. Salaries and benefits for clinicians and support staff, billing and administration, equipment, supplies, medication, and square footage related to care delivery are among the costs that should routinely be monitored.^5–6

Studies of factors affecting healthcare organizations’ transformation-related costs have generally concerned hospitals.⁷ However, a few studies use primary care organizations as the unit of analysis. Based on the opinions of focus group participants, Reiter and colleagues⁸ describe the costs associated with Patient-Centered Medical Home (PCMH) transformation as falling into two categories: costs of personnel time for transformation-related activities, and non-personnel resource costs such as for IT and supplies. Of the few studies that quantify costs of organizational transformation, Nocon and colleagues⁹ examined the relationship between PCMH adoption and operating costs of Federally Qualified Health Centers (FQHCs). Using covariates such as annual number of patient visits, annual number of patients served, and number of physician FTEs, the study found that FQHCs with a greater number of PCMH attributes to have higher operating costs. To address these gaps in the literature, we analyze the costs associated with the practice transformation of joining an ACO.

Methods

This study analyzes secondary data related to costs of RHC ACO participants. It is one of several research activities of a larger project that examines the impact of ACO participation and other factors on preventive care effectiveness and cost efficiency of RHCs. IRB approval to conduct the research was obtained by the University of Central Florida.

Unit of Analysis

The unit of analysis for this study was the individual RHC. We compared the costs of those RHCs that were Medicare Shared Savings Program (MSSP or SSP) ACO participants during 2012 and/or 2013 to a similar group of RHCs that did not participate in this type of ACO during the same year(s.)

Data Sources

There were three principal sources of secondary data: the Medicare Cost Report, the CMS Chronic Condition Data Warehouse or CCW, and the Area Health Resource File or AHRF System. Additional sources include the CMS Online Survey and Certification Reporting System (OSCAR), the National Minimum Data Set (MDS). All secondary data were collected for the period 2007 – 2013.

The Medicare Cost Report is an annual cost report that Medicare-certified RHCs are required to submit to CMS or a Medicare Administrative Contractor. Although the cost report is limited in regard to some details of clinic operation, it is the most complete source of available data for RHCs at this time. It contains provider-specific data such as facility characteristics, cost and charges by cost center, utilization data, and financial statement data.

The CCW data are patient-specific. The data were linked to specific Medicare and dual-eligible patients affiliated with a particular RHC.¹The AHRF (formerly ARF) was the main source of data for context (community or demographic) variables. The AHRF is a health resources information system that compiles sociodemographic, economic, and health-related data at the county level.¹⁰

Measurement Variables

Our analyses were conducted using a “treatment effects approach.” Within this approach, there are two key variables: the “treatment” and the “outcome.” The treatment in our study is participation in an ACO. We operationalize this as the binary variable ACOstart12 that equals 1 if an RHC joined an ACO in 2012 and equals 0 if an RHC was not in an ACO in 2012. Likewise, RHCs that joined ACOs in 2013 were indicated with the binary variable ACOstart13. The outcome in this study is average annual cost per visit for each RHC in 2012 and 2013. It was calculated as follows: total allowable cost excluding vaccine divided by total adjusted visits. Once we had calculated this variable, we plotted it and found clear evidence that it was skewed, as is often true for cost variables. As is commonly done for skewed dependent variables in regressions, we transformed this variable by taking the natural logarithm of each value. This makes the distribution of the transformed variable become normal.

In order to estimate the “treatment effect” on cost/visit of an RHC joining an ACO, we must specify a model with covariates that are related to the binary treatment variable ACOstart12. We turned to the relevant literature for guidance about what covariates belong in that model.^11,12 Based on the sparse literature, our treatment model contained the following covariates that were available to us in our sample:

1. TotFTE: Size as measured by total FTE

2. Provbsd: A binary variable that = 1 if Provider-based RHC and = 0 if Independent RHC

3. Control: Type of control of RHC according to 1 of 9 classifications, ranging from for-profit to government controlled

4. Age: The number of years the practice has been RHC certified

5. Rural: A binary variable that equals 1 if the RHC is in an isolated rural location and = 0 otherwise

Sample

We created a sample using data from the dataset created for the larger project that examines the impact of ACO participation and other factors on preventive care effectiveness and cost efficiency of RHCs. This dataset contains data on a panel of clinics that were continuously certified as RHCs throughout the years 2007 – 2013. All clinics were from the DHHS Region 4, which includes Kentucky, Tennessee, North Carolina, South Carolina, Georgia, Florida, Mississippi, and Alabama. This entire panel of over 800 RHCs was the starting point for this analysis. Since the first year that an RHC could join an ACO was 2012 and our most recent year of complete data is 2013, we restricted our analysis to include only 2012 and 2013 data. Therefore, our analysis sample contained RHCs in a MSSP ACO in 2012 and/or 2013 and those not in an ACO in either year. This gave us a sample of 821 non-ACO RHCs and 20 RHCs that were in MSSP ACOs.

Analytical Approaches

There are several methods for estimating treatment effects from observational data. First, taking the difference between the sample means for the treated and untreated subjects seems a natural approach to take. However, we should not estimate the treatment effect this way because there may be covariates that are related to both the outcomes and the treatment. Second, regression is widely used. However, “A common problem arises in observational studies when treated subjects (e.g., ACO RHCs) are not well matched to controls (non-ACO RHCs) on factors thought to be correlated with treatment. When that happens, covariate control through regression can produce unreliable estimates even if all relevant variables are observable.”¹³ Another problem with regression arises from the necessity to specify a functional form for how treatment impacts the outcome. Different functional forms can give different estimated treatment effects.

A better alternative for estimating treatment effects is to use one of the several weighting or matching estimators that are widely used in the treatment effects literature. Propensity score matching (PSM) is one of the most popular and is appropriate to use in this analysis. Therefore, we employed this estimator for the following reasons. First, once we specify enough of these covariates that are related to the treatment, any remaining influences on the treatment should not be related to the outcomes. Second, since the propensity scores are estimated without an outcome variable model, it is unnecessary to make any assumptions about the functional form by which treatment impacts the outcome of interest.²

Matching each treated subject with one control subject is, by far, the most common method used in the treatment effects literature. However, you can match each subject with multiple subjects of the opposite treatment level. It is important to be clear about the benefits and costs of multiple matching. First, more matches per subject uses more of the information (i.e., more observations) in the sample. Second, matching on more distant neighbors can reduce the variance of the estimator (i.e., increase its efficiency).¹⁴ However, the cost of multiple matches is a possible increase in bias. As you match more controls with each treated subject, you are finding successively poorer matches.

Given this background, we will present results for five different levels of matching, 1:1, 1:2, 1:3, 1:4, and 1:5. We will then examine the results to see how similar or how different the estimated treatment effects are. This sensitivity analysis might enable us to decide whether to trust the 1:1 matching results (based on the best single matches) or the 1: k results (with the advantages of using more information and giving more efficient estimates.)

Last, there are a number of treatment effects measures that we might estimate. However, the two most common measures are the average treatment effect (ATE) and average treatment effect on the treated (ATET.) The ATE is the mean of the difference between actual and potential outcomes (the outcome for the matched subject) for all subjects – those that received the treatment (e.g., joined an ACO) and those that did not. The ATET is the mean difference between actual and potential outcomes among only the subjects that actually received the treatment (e.g., joined an ACO). Among others, Heckman^15–18 argues that in a variety of situations, it is the ATET that carries the greatest weight. The essence of his argument is that in deciding if a policy or proposal is beneficial, the interest is not whether the initiative is beneficial for all subjects on average but whether it is beneficial for those subjects who are treated or who would choose the treatment. We will report the ATE results because they are what most studies describe. However, our focus in this study is on those RHCs that are in an ACO or might join one. Consequently, we will also report results for the ATET.

Findings

The results are presented in five parts: A. mean cost per visit before and after joining an ACO (Table 1); B. tests of balance on treatment model covariates before matching (Table 2); C. tests of balance on those same covariates after matching (Table 3); and D. treatment effects estimation results using the PSM estimator for ATET (Table 4) and ATE (Table 5.)

Table 1.

Mean Cost/Visit for Two Groups of RHCs in 2011 and 2012.

(A) Mean cost/visit for two groups in 2011: RHCs that joined an ACO in 2012 and those that did not
	Mean	[95% Confidence Interval]
Did join	$110.97	$82.77–$139.16
Did not join	$111.26	$108.09–$114.42

(B) Mean cost/visit for two groups in 2012: RHCs that joined an ACO in 2012 and those that did not
	Mean	[95% Confidence Interval]
Did join	$90.05	$59.71–$120.39
Did not join	$117.57	$114.10–$121.04

Table 2. Independent two-sample tests of differences, two tests.

ACO RHCs vs. non-ACO RHCs, pre-PSM

(See notes for details)

Variable	mean or proportion, ACO RHCs	mean or proportion, non-ACO RHCs	P-value	test type
TotFTE	4.858	3.148	0.000	1
Control07	4.184	4.065	0.480	1
AgeRHC	7.245	7.457	0.667	1
Provbsd	0.429	0.336	0.022	2
Rural	0.755	0.807	0.121	2
stateAL	0.000	0.071	0.001	2
stateCA	0.224	0.144	0.008	2
stateFL	0.327	0.140	0.000	2
stateGA	0.041	0.090	0.041	2
stateKY	0.041	0.151	0.000	2
stateMS	0.184	0.160	0.459	2
stateNC	0.082	0.092	0.660	2
stateSC	0.041	0.099	0.020	2
stateTN	0.061	0.053	0.652	2

Note 1:

TotFTE: RHC Size, total FTE

Control07: 9 types of control, ranging from for-profit to government controlled

AgeRHC: number of years practice has been RHC certified

Provbsd: = 1 if Provider-based RHC & = 0 for Independent RHC

Rural: = 1 if RHC in an isolated rural location

each stateXX: = 1 if RHC in that state

Note 2:

test 1: t-test, H0: means are equal across two categories (interval variable)

test 2: Pearson’s chi-squared, H0: proportions are equal across two categories (binary variable)

Table 3. Independent two-sample tests of differences, two tests.

ACO RHCs vs. non-ACO RHCs, post-PSM

(See notes for details)

Variable	mean or proportion, ACO RHCs	mean or proportion, non-ACO RHCs	P-value	test type
TotFTE	4.858	3.144	0.061	1
Control07	4.535	4.287	0.220	1
AgeRHC	6.663	7.478	0.165	1
Provbsd	0.584	0.445	0.086	2
Rural	0.832	0.812	0.606	2
state AL	0.000	0.075	0.000	2
stateCA	0.317	0.168	0.312	2
stateFL	0.168	0.116	0.175	2
stateGA	0.059	0.084	0.329	2
stateKY	0.059	0.177	0.450	2
stateMS	0.228	0.166	0.153	2
stateNC	0.069	0.071	0.947	2
stateSC	0.030	0.092	0.231	2
stateTN	0.069	0.051	0.488	2

Note 1:

TotFTE: RHC Size, total FTE

Control07: 9 types of control, ranging from for-profit to government controlled

AgeRHC: number of years practice has been RHC certified

Provbsd: = 1 if Provider-based RHC & = 0 for Independent RHC

Rural: = 1 if RHC in an isolated rural location each stateXX: = 1 if RHC in that state

Note 2:

test 1: t-test, H0: means are equal across two categories (interval variable)

test 2: Pearson’s chi-squared, H0: proportions are equal across two categories (binary variable)

Table 4.

Estimated ATET - PSM estimator, M = 1 – 5

Variable	M = l	M = 2	M = 3	M = 4	M = 5
ACO (1 vs 0)	18.577	18.614	16.946	16.170	15.002
	6.982	6.636	6.438	6.213	6.170
	0.008	0.005	0.008	0.009	0.015

The mean ATET is $17.06 per visit.

Table 5.

Estimated ATET - PSM estimator, M = 1 – 5

Variable	M = l	M = 2	M = 3	M = 4	M = 5
ACO (1 vs 0)	14.498	11.408	13.225	15.328	14.119
	6.008	4.488	4.175	4.084	3.916
	0.016	0.011	0.002	0.000	0.000

The mean ATE is $13.72 per visit.

We once again emphasize that these results are for a sample containing only 20 ACO RHCs. So, we hope it is clear that we are presenting results that must be considered very preliminary. Whatever conclusions we draw from these results are intended to merely suggest what might be found once many more RHCs join ACOs.

A. Mean cost per visit before and after joining an ACO.

As a first level of analysis, we determined the mean cost per visit for two groups of RHCs: those that joined a MSSP ACO and those that did not. Costs were determined for each of two years prior to when the RHC joined a MSSP ACO: 2011 (for RHCs that joined or did not join in 2012), and 2012 (for RHCs that joined or did not join in 2013.) (See Table 1.) We did this to investigate whether higher cost RHCs joined ACOs in order to derive cost savings from improved care coordination. The implication of finding this is true would be that higher post-ACO costs might, in part, be due to higher pre-ACO costs. The 95% confidence intervals for both groups (ACO RHCs and non-ACO clinics) in Table 1 overlap in 2011 and in 2012. Therefore, we determine that there is not strong evidence that higher cost RHCs joined ACOs. So, we can conclude that any higher post-ACO costs were probably not due to higher pre-ACO costs in the RHCs that joined ACOs.

C. tests of balance on those same covariates after matching.

Covariates are balanced (at least have the same means) in experimental data because treatment assignment is independent of the covariates, due to the study design. In contrast, covariates must be balanced by weighting or matching in observational data for two reasons, both of which can bias the estimated treatment effect if not accounted for in the analysis. First, treatment assignment may be related to the covariates that also affect the outcome of interest. Second, if the covariates are not balanced after the PSM estimation, we have not sufficiently accounted for differences between the two groups (treated and not treated) that can confound the estimate of the treatment effect. So, if the covariates are not balanced across the treatment groups, we should be skeptical of the results. In fact, one of the pioneers of propensity score matching, Donald Rubin,¹⁹ recommends finding a model that balances the covariates before looking at results for the estimated treatment effect. We agree with this position. So, we will present results about covariate balance for both before and after matching. Consistent with Rubin’s belief, we will only present our treatment effects estimates after we find that propensity score matching did, indeed, balance the covariates in the treatment model.

Table 2 shows the results of covariate balance checking (for means) for the measures in the treatment model (the covariates related to the binary treatment variable.) These variables are defined in the Measurement variables part of the Methods section. In this table, we see that the means of the treated (ACO) RHCs and the untreated RHCs are significantly different from each other (most p-values are 0.05 or lower for the H₀ of equal means) before matching. After matching, Table 3 shows that we cannot reject the H₀ of equal means across treatment status for 13 of the 14 covariates (for these, the p-values range from 0.061 to 0.947.) This satisfies Rubin’s¹⁹ requirement about finding a treatment model that balances its covariates after matching. Now, we can have confidence that we have eliminated that source of confounding of our estimated treatment effects.

D. Treatment effects estimation results using the PSM estimator for both ATE and ATET

Table 4 displays the results of estimating ATET; Table 5 shows those for estimating ATE. The first numerical row contains the average treatment effect of joining an ACO: the change in cost/visit. The second row shows the standard errors of those estimates, and the third row displays the p-values for testing the H₀: that the average treatment effect is zero in the population. The different columns present results for different numbers of matches for each treated (ACO) RHC. The M = 1 column is for the case of finding one non-ACO RHC to match each ACO RHC. The M = 2 is for two matches per ACO RHC. The remaining three columns are for 3, 4, and 5 matches.

Note that the standard errors monotonically decrease as the number of matches rises for the ATE estimates. This means that we are seeing considerably more precise estimates of the treatment effect with 5 matches than with one. This is the chief advantage of finding more than only one match per treated RHC.

The estimates of the average treatment effect on the treated (ATET) pertain to only those RHCs that joined ACOs. The results show that those 20 sample ACO RHCs experienced an average from $15.00 to $18.61 higher cost/visit than the matching non-ACO RHCs. The estimates of the average treatment effect (ATE) refer to every one of the RHCs in the entire sample - those that joined an ACO and those that didn’t but regarded as if they did. The results show that those 828 RHCs in the entire sample RHCs experienced, on average, from $11.41 to $15.33 higher cost/visit when in an ACO than when not. All the p-values allow us to conclude that the effect on cost/visit is not zero in the population because those p-values range from 0.00 to 0.016.

Discussion

Primary care providers are an essential component of the current and future health care delivery system. Although there is an expectation that some of these providers will choose to transition to become members of the newer models for health care delivery such as ACOs, there is little knowledge about the effect of such practice transformation on their costs. The intent of this study was to discover whether or not transformation of primary care organizations to become ACO participants had an effect on their costs and, if so, the amount of those costs.

Our findings suggest that RHCs in MSSP ACOs experienced a higher cost per visit than their non-ACO counterparts during the first two years of participation. These findings held true when both the ATET and the ATE approaches were used. Specifically, the mean increase in cost/visit (across all five estimates) is $17.06 per visit for the ATET results and is $13.72 per visit for the ATE results.

This study has one clear limitation, however. Recall that the ATET is the mean of the difference between actual and potential outcomes among only the subjects that actually joined an ACO. Our study analyzed data from the first two years of the Medicare Shared Savings Program ACOs, which may explain why there were only 20 ACO RHCs in our sample. So, even when we’ve matched 5 control RHCs with each of the 20 treated RHCs we have only 120 observations (out of over 800) used to estimate the average treatment effect on the treated (ATET) of joining an ACO.³ Counterbalancing this small sample size for the ATET results is the fact that for the ATE estimates we use the entire sample of over 800 RHCs.

Despite the limitations inherent to working with a small sample of RHCs in ACOs, the findings reported here are valuable in that they provide some of the earliest indications of the cost structure of RHCs that choose to participate in one type of ACO, or be more inclined to do so. Much of the literature to date describes ACOs as a whole, rather than the factors that are associated with an organization’s being part of an ACO. When organizational characteristics are examined, quite frequently they concern hospitals. For example, in one of the few studies of rural ACOs, Huff²⁰ found that rural hospitals that participate in Medicare ACOs have a long-standing relationship with local doctors, have experience mining their EHR systems for financial and patient treatment patterns, and have practiced several approaches to minimizing hospital admissions. The motivations for primary care providers, particularly those located in rural areas, to join ACOs are likely to be quite different from those of hospitals. Small rural providers located in more remote areas, in particular, may feel that they have less opportunity to join ACOs, minimal opportunity to provide leadership in them, and/or less well prepared – financially and in terms of IT infrastructure – to be able to meet requirements for collecting, organizing, and sharing data with ACO partners.

As ACOs progress to become a more common part of the healthcare delivery system, knowledge about ACOs and about primary care providers’ experiences with them, will be widely dispersed. As familiarity with the ACO concept grows, so may rural primary care providers’ willingness to participate in this model of healthcare delivery. Based on a survey of RHC managers, Wan and colleagues²¹ found that RHCs were more willing to join ACOs if they were knowledgeable about ACOs or if they perceived a benefit in joining ACOs, such as the potential for improving the quality of health of their patients and their communities.

Creating successful ACOs requires developing a new culture within the ACO member organizations and throughout the ACO as a whole. ACOs are a value-based model of healthcare delivery. Thus, many prospective ACO member organizations will need to undergo fundamental cultural change from one that focuses on volume to one that focuses on the quality of the services they provide. Performance measures will be evaluated and modified to reflect that change. Successful ACOs must strengthen their health information technology infrastructure, develop new systems, and devise approaches for sharing savings among their participant organizations.²² Processes to coordinate care among services and care teams must be established or improved.

The changes involved in joining an ACO are time-consuming and costly. Recent interviews with management of RHCs that are participating in ACOs revealed that the start-up costs of establishing, maintaining, and sustaining the necessary ACO infrastructure may deter RHCs from joining ACOs.²³ However, many personnel from RHCs that are not currently ACO participants reported that they are not yet familiar with ACOs, and indicated that they are interested in learning about the experiences of primary care providers that become ACO participants.¹¹

Conclusions

By means of this longitudinal study of a panel of RHCs, we address the need for greater knowledge about the cost of practice transformation for the individual ACO primary care participant. At this very early stage of ACO development, our results must be considered very preliminary. Whatever conclusions we draw from these results are intended to merely suggest what might be found once many more RHCs join ACOs. Having said that, the conclusions we draw from this early analysis can still be interesting and lay a foundation for more analysis after data are available when more RHCs join ACOs. As ACOs become more widespread throughout the U.S., providers, government agencies, and insurers alike will benefit by learning about the experiences of early ACO adopters such as the clinics described in this study.