Performance-Based Contracting Within a State Substance Abuse Treatment System: A Preliminary Exploration of Differences in Client Access and Client Outcomes

Abstract

To explore whether the implementation of performance-based contracting (PBC) within the State of Maine’s substance abuse treatment system resulted in improved performance, one descriptive and two empirical analyses were conducted. The first analysis examined utilization and payment structure. The second study was designed to examine whether timeliness of access to outpatient (OP) and intensive outpatient (IOP) substance abuse assessments and treatment, measures that only became available after the implementation of PBC, differed between PBC and non-PBC agencies in the year following implementation of PBC. Using treatment admission records from the state treatment data system (N=9,128), logistic regression models run using generalized equation estimation techniques found no significant difference between PBC agencies and other agencies on timeliness of access to assessments or treatment, for both OP and IOP services. The third analysis, conducted using discharge data from the years prior to and after the implementation of performance-based contracting (N=6,740) for those agencies that became a part of the performance-based contracting system, was designed to assess differences in level of participation, retention, and completion of treatment. Regression models suggest that performance on OP client engagement and retention measures was significantly poorer the year after the implementation of PBC, but that temporal rather than a PBC effects were more significant. No differences were found between years for IOP level of participation or completion of treatment measures.

Introduction

In the USA, single state agencies (SSAs) are the federally designated public administrative entities responsible for the coordination of substance abuse services in states and territories. A primary responsibility of SSAs is to provide substance abuse treatment services for clients who lack the means to purchase services. Most SSAs do not provide services directly; they purchase services from community-based systems of care. SSAs are financed by a combination of federal, state, and local funding streams and have traditionally relied on three contracting strategies to purchase community-based treatment services: cost reimbursement, fee-for-service and prospective payment systems.

Under cost reimbursement systems, SSAs may solicit bids from treatment service agencies to provide a pre-determined set of services. Providers then submit required cost documentation to the purchaser on either a monthly or quarterly basis. The purchaser reimburses accordingly, up to the amount agreed upon in the initial bid. Fee-for-service contracts allow providers to be reimbursed for an unspecified amount of services rendered so long as the service is consistent with defined care practices. Prospective payment contracts set reimbursement amounts for particular services in advance of service delivery, regardless of the actual cost to the service provider. Payments can vary with patient characteristics or other factors that affect costs. SSAs use any number of strategies within these contracting options to attempt to ensure treatment quality, including mandating treatment protocols and staffing patterns, and enforcing physical plant specifications. However, such procedural requirements do not necessarily assure that quality treatment will result.^1,2

Performance-based contracting (PBC) is one method that can be used to improve quality. PBC offers direct financial incentives to health and human service providers, contingent upon the achievement of pre-determined levels of performance on defined indicators. While there are many variations, most often government purchasers provide a funding base that can be supplemented with financial incentives that are tied to performance. Despite the endorsement by the Institute of Medicine for the use of performance measures in substance abuse treatment,^3,4 SSAs have been slow to engage in PBC.

The recent implementation of PBC within the publicly funded substance abuse treatment system in the State of Maine provides an opportunity to begin to examine the relationship between agency participation in PBC and performance on access and retention measures. PBC was implemented first and foremost as a performance measurement tool to enhance the data-driven decision-making capabilities of the state substance abuse agency, the Office of Substance Abuse (OSA). Given this reality, certain data limitations, explained more fully in later sections of this article, dictate the types of research analyses that can be conducted at this early stage of implementation. Still, the rationale for conducting this study is strong as the behavioral health care field needs to continually assess its efforts to achieve higher quality outcomes while controlling costs.

The analysis of available administrative data can add to the growing literature on the evolution of PBC within the behavioral health care system in unique ways. First, detailed information about the implementation of a new PBC system will be shared, including impacts on utilization and payment structure. States seeking to either enhance existing PBC systems or implement new PBC systems will find this information of practical use.

Secondly, this article will add to the theoretical base about PBC. This article will examine whether performance on time to assessment and time between assessment and treatment for outpatient (OP) and intensive outpatient services (IOP) differed for agencies that became a part of the PBC system. This research will also examine whether level of participation, retention, and completion of treatment among admissions to agencies that implemented PBC during SFY08 differed significantly from comparable client outcomes among admissions that occurred in the year prior to the implementation of PBC (SFY07). Based on preliminary results from these lines of inquiry, policy implications can be explored and future directions for research can be suggested.

Implementation of PBC in Maine

The Maine Office of Substance Abuse (OSA) introduced a version of PBC in 1992 across all of its treatment modalities in an attempt to shift the publicly funded treatment system from a focus on outputs to a focus on outcomes. Organizationally, the information gathered through this effort was helpful in providing data to support contract management decisions. Empirically, however, the results of this initial iteration of PBC were mixed; providers reduced the amount of some services provided but reported better outcomes for certain measures.⁵ Analyses of the PBC system also raised issues of possible adverse selection and gaming practices, whereby client-level information entered into the state data reporting system used for PBC reporting was found to be inconsistent with agency medical records.^6–8 The desire to address these criticisms of the PBC system, combined with the development of and knowledge spread about standardized performance measures throughout the country, helped propel the Maine SSA towards a broad restructuring of its PBC system. Maine implemented changes to its treatment provider contracts in SFY 2008 (July 1, 2007–June 30, 2008), focusing the new PBC effort on ambulatory services for the first year of implementation.

Any agency that had an existing contract with OSA for OP or IOP services was included in the first year of PBC. The goal of the new system was to foster efficiency and quality of service within the treatment system by tying performance to actual payment level. The new system attempted to correct for the aforementioned concerns and to institutionalize an organizational emphasis on improving access to and retention in substance abuse treatment. The new Maine contracts specified base amounts, potential incentive amounts, and possible financial penalties. To encourage efficiency, agencies that exceeded 100% of contracted units of service per quarter received an incentive payment of 5% of the calculated quarterly payment. Agencies that did not meet 90% of the contracted service units for the quarter received a cut in reimbursement of 5% for that quarter. The intent of this utilization incentive was to ensure that agencies did not in any way limit the number of clients served.

As part of this system, four access and retention measures were set for OP and IOP contracted agencies. OP and IOP services were assigned performance targets, in terms of median number of days, from time of first contact (i.e., date of first phone call to an agency) to assessment and from time of assessment to actual participation in treatment. For example, agencies providing OP services were required to have a median time to assessment of five calendar days and a median time to treatment of 14 calendar days. For IOP services, median times to assessment and treatment were set at four calendar days and seven calendar days.

In addition, OP services were measured on two retention related measures—participation in at least four treatment sessions and length of stay of 90 days or more. Agencies were required to have at least 50% of their OP admissions stay for four or more days and to have at least 30% of their OP admissions stay for 90 days or longer. IOP services were measured on two retention related measures, as well—participation in at least 4 days of treatment and completion of treatment. A minimum of 85% of IOP admissions were required to stay at least four sessions, and a minimum of 50% of IOP admissions were required to complete treatment.^*

Performance in meeting all of these goals would affect agency eligibility for incentive payments, base contract payments, and financial penalties. In total, a program stood to gain or to lose 9% of its contract amount under the new system. Payment adjustments (either incentives or penalties) were made during the middle month of the subsequent quarter, based on performance during the measured quarter.

The new system was designed to be an improvement over the prior PBC system in several ways. First, the new PBC system included concrete incentives and disincentives that would be more difficult to artificially manipulate. Provider accountability was improved, and contract management decisions were now truly data driven. While the prior system, in theory, tracked 15 measures for OP and 14 measures for IOP, funding decisions were not specifically tied to performance.^5–7 As the number of concrete performance measures for OP and IOP services were reduced substantially from the prior system, better clarity and focus were available to both treatment provider and state agency staff. In addition, the level of discretion available to state agency contract managers was reduced as incentives and penalties were set as fixed percentages of overall contract amounts and performance targets were clearly spelled out. The timeliness of payment adjustments was increased as well. In the new PBC system, payment adjustments were applied quarterly rather than annually.

Methods

Data

For the first, descriptive analysis, administrative data were used to summarize payment adjustments made and utilization rates for the year following implementation of PBC. Agency-level information was obtained from program fiscal coordinators at the state substance abuse agency, as well as from the business intelligence system mentioned above.

For the second and third analyses, encounter-level data were extracted from TDS as Excel files and converted into SPSS files and SAS files for statistical analyses.

Data related to times to assessment and times to treatment was only available beginning in SFY08. Data gathered for 1 year, for two groups of agencies (those in the PBC system and those not in the PBC system), were used.

As data related to level of participation, length of stay, and completion of treatment were available both prior to (SFY07) and after (SFY08) the implementation of PBC, 2 years’ worth of data were used. These data were limited to those agencies that became a part of the PBC system in SFY08. The same agencies participated in PBC during SFY07 and SFY08.

Measures

Times to assessment and treatment

Within TDS, time to assessment is calculated as the difference, in days, between the date of first phone call and the date of the first face-to-face meeting with the client. As defined within the PBC system, the first face-to-face meeting is when a client clinical assessment should occur. Time to treatment is also calculated as a continuous variable indicating the difference between the date of the first face-to-face meeting with the client and the date of the first treatment session. The time to assessment and time to treatment data were recoded to remove any possible errors, restricting cases to those that had positive values between zero and 364. Time to assessment and time to treatment were included in the dataset as both continuous variables to measure the actual number of days and dichotomous variables to measure achievement of the performance metrics. The continuous variables were used for the descriptive analyses. For the logistic regressions, binary variables were used, indicating whether a particular episode of care met or did not meet the standards defined within the PBC system.

Participation, retention, and completion of treatment

For this portion of the analysis, outcomes are defined as number of treatment sessions attended, total length of stay in treatment, and completion of treatment. One OP treatment session is measured as a 1-h counseling session. For IOP services, a session equates to 1 day of treatment. Length of stay is calculated as the difference, in days, between admission and discharge. Discharge date is recorded as the last face-to-face meeting with the client. These outcome variables were included as either continuous or binary variables in different analyses. The continuous variables were used for the descriptive analyses. For the logistic regressions, binary variables were used, indicating whether a particular episode of care met or did not meet the standards defined within the PBC system.

Completion of treatment was used as a dichotomous variable in all analyses, based on a definition from the Maine Office of Substance Abuse.

A client achieves at least two thirds of his/her most current agreed upon treatment plan, and the clinician is in agreement with the discharge. The plan should include objectives specific to client need and might include the following:

• Abstinent during treatment

• Significant reduction in problem use

• Willingness to voluntarily seek continued care as necessary

• Participation in self-help⁹

The definition of completion of treatment as described here is not only used for performance measurement within Maine but is also reported to federal funders for inclusion in the national Treatment Episode Data Set.

Analytic plan

Times to assessment and treatment

The analysis of the access measures uses cross-sectional data from the year following the implementation of PBC, as the measurement of time to assessment and time to treatment only became available in the year that PBC was implemented. Descriptive analyses were run first to determine whether any association existed between the attainment of access measures and participation in PBC. Logistic regression models were run for each model, using generalized estimating equation techniques. Meeting the time to assessment or treatment measures for OP or IOP services were included as binary dependent variables.

Since the dependent variables were dichotomized (i.e., met access standards), logistic regression estimates were applied within the generalized estimating equation framework. This approach was appropriate because the admission and discharge data were clustered within different treatment programs. Client outcomes may thus be correlated within programs. Because individual level data were contained within agencies, autocorrelation of respondents would violate the assumption of independence for logistic regression analysis, potentially shrinking standard errors and increasing type I error rates. Generalized estimating equations were used to address these issues.

Age was included as a binary independent variable, with a value of one indicating older adults. Persons younger than 18 were not included in the data set. Gender and race were included as covariates, with a value of one indicating males and whites. Marital status was included as well with a value of one indicating clients were either married or cohabitating. Employment status was included as a value of one indicating full-time employment, as this was thought to possibly impact the ability to quickly attend an appointment. Admissions with a co-occurring mental illness were indicated by a value of one on the mental illness variable. Insurance status was included as a proxy for systems level variables that have been found to influence access to treatment. A value of one was assigned to admissions with no insurance. Admissions initially referred from criminal justice sources were assigned a value of one for the criminal justice variable. Agency size was categorized into two groups: small/medium (assigned a value of zero) indicated that an agency had fewer than 500 ambulatory admissions over the SFY07 and SFY08 period. Large agencies (assigned a value of one) admitted 500 or more episodes of ambulatory care during that time period.

Logistic regression techniques were used to control for the possible impact of these covariates on the binary dependent variables.

Participation, retention, and completion of treatment

The analysis of the retention measures uses cross-sectional data from 2 years, 2007 and 2008, to examine differences in service encounter outcomes. As number of treatment sessions were completed, length of stay and completion of treatment may be influenced by a variety of individual client and organizational characteristics that might have changed from year to year,¹⁰ and a number of covariates were included in the analyses. The covariates primarily mirror those included in the prior regressions. Interaction terms were added, however, to attempt to decipher effects that might have been time dependent. Interaction terms were included for older adults and 2008, gender and 2008, mental illness and 2008, and criminal justice referral and 2008. Series of logistic regression models were run, following the specifications mentioned above, using a difference in differences framework. Variables for performance-based contracting cohort, year, and an interaction variable for performance-based contracting and year were included. Agencies that became part of the performance-based contracting system in the second year were coded as ones, as were episodes admitted during the second year. The interaction term would provide a value of one for those episodes that were admitted to performance-based contracting agencies during the second year.

Logistic regression models were run, following the specification below:

$$\text{Prob}(\text{Focal variable})=F({\beta }_0+{\beta }_{1}R+{\beta }_{2}S+{\beta }_3(R\times S)+{\beta }_{4}X+{\beta }_{5}V)$$ (1)

where F is the logistic cumulative distribution function, R is the indicator of agency year, S is the indicator of the eventual PBC cohort of agencies, and (R×S) is the interaction variable of year by PBC status; vectors X and V contain the control variables, where vector X contains age group, gender, race, marital status, co-occurring mental illness, criminal justice involvement, primary drug of abuse, employment status, and presence of health insurance with vector β₄ containing corresponding coefficients; and vector V contains agency size, with vector β₅ containing corresponding coefficients.

Results

Payment and utilization information

Summary of payments received

Over the course of the entire fiscal year, the 17 participating agencies were budgeted to receive $3,531,364 in base funding and had the potential to gain $238,099 if baseline performance targets were surpassed. Only a portion ($44,839 or 18.83%) of the incentive money was utilized, however. For the year, payment adjustments ranged from −6.50% to +6.75% per agency. Among agencies, the mean payment adjustment was 0.01%(s=3.82), and the median level of payment adjustment was −1.13%.

Summary of units of service provided

OP services were contracted to provide 254,790 quarter hours of service during SFY08. As a whole, 244,897 OP units or 96.12% of contracted units were delivered. IOP services were contracted to provide 18,026 days of service. During SFY08, 20,193.36 days of service (or 112.03% of contracted IOP units) were provided.

Times to assessment and treatment

Sample

Table 1 shows a detailed breakdown by service setting for admissions demographics, agency size, and access performance measures for the sample used for the study related to access measures. The data set included only those adult admissions from SFY08 that were self-identified as having a primary problem of substance abuse.

Table 1.

Adult treatment admission demographics (SFY08)

	OP		IOP
	PBC	Non-PBC	PBC	Non-PBC
	(N=3,915)	(N=2,745)	(N=1,156)	(N=1,312)
	%	%	%	%
Age 18–25	25.0	20.8a	24.9	24.3
White	93.8	94.8	94.7	95.3
Male	55.9	66.2a	55.3	58.5
Married/cohabitating	22.1	28.9a	23.4	24.3
Primary drug: alcohol	50.8	64.4a	39.3	40.7
Primary drug: marijuana	12.0	9.7a	9.1	4.2a
Primary drug: cocaine	8.1	5.1a	9.1	8.0
Primary drug: opiates	26.9	19.2a	40.9	44.1
Co-occurring MI	47.4	63.7a	64.4	59.5a
Referred from CJ	26.9	15.7a	26.6	12.4a
Full-time employment	25.6	39.3a	19.8	27.2a
No health insurance	31.2	29.5	24.5	19.7a
Admitted to large agency	79.5	9.5a	82.8	78.1a
Met time to assessment (PBC target)	60.8 (5 days)	52.3a	69.2 (4 days)	78.1a
Met time to treatment (PBC target)	91.7 (14 days)	84.8a	85.9 (7 days)	92.8a

^aPBC vs non-PBC, p<0.05

Descriptive analyses

As mentioned previously, time to assessment and time to treatment were included in the data set as both continuous and discrete variables. Table 1 also contains information about the percent of agencies meeting these waiting time measure. Using the binary variables, 57.3% of OP admissions met the minimum standard for time to assessment. PBC agencies performed significantly better, (X²=56.310, df=1, p=0.000). More OP admissions met the time to treatment standard (91.6%). Again, PBC agencies performed significantly better (X²=83.360, df=1, p=0.000).

Among all IOP admissions, 73.4% met the time to assessment measure and 89.1 met the time to treatment measure. Significantly more non-PBC admissions met both the time to assessment standard (X²=53.360, df=1, p=0.000) and the time to treatment standard (X²=67.245, df=1, p=0.000).

The results of independent sample t tests on average times to assessment and treatment for PBC and non-PBC admissions, based on using the continuous variables, are shown in Table 2. When comparing the mean number of days, results suggest that PBC agencies had significantly lower average number of days to assessment and treatment for OP admissions than non-PBC agencies. The mean time from first phone call to assessment for all OP admissions was 8.08 days. PBC admissions had an OP face-to-face encounter in an average of 7.08 days, while non-PBC admissions had to wait nearly 10 days for their first assessment (9.70 days). Times between assessment and treatment were shorter, with all admissions having a mean wait of 4.89 days between assessment and treatment. OP admissions to PBC agencies had a significantly lower mean wait time (4.38 days) than admissions to non-PBC agencies (5.70 days).

Table 2.

Average time to assessment and treatment (SFY08)

	All	PBC	Non-PBC	t test
Average time to OP assessment	M=8.08	M=7.08	M=9.70	p=0.000
	N=6,629	N=4,077	N=2,552
	Range, 0–336	Range, 0–231	Range, 0–336
	SD, 16.372	SD, 12.988	SD, 20.559
Average time to OP treatment	M=4.89	M=4.38	M=5.70	p=0.000
	N=6,629	N=4,069	N=2,533
	Range, 0–209	Range, 0–209	Range, 0–188
	SD, 12.325	SD, 11.841	SD, 13.026
Average time to IOP assessment	M=7.91	M=8.24	M=7.51	p=0.500
	N=2,457	N=1,348	N=1,109
	Range, 0–291	Range, 0–288	Range, 0–291
	SD, 26.395	SD, 24.163	SD, 28.882
Average time to IOP treatment	M=3.89	M=5.43	M=2.02	p=0.000
	N=2,459	N=1,348	N=1,111
	Range, 0–317	Range, 0–317	Range, 0–146
	SD=14.939	SD, 18.727	SD, 7.892

No significant difference existed between mean numbers of days to assessment for IOP admissions. Overall, IOP admissions had their first face-to-face contact within 7.91 days of their first phone call to an agency. For time to treatment for IOP admissions, PBC admissions had significantly higher average numbers of days. PBC admissions to IOP waited 5.43 days between assessment and treatment, while non-PBC admissions to IOP experienced a significantly lower wait time of 2.02 days.

Regressions

As logistic regression analyses suggest that there were no significant effects of PBC on OP or IOP time to assessment or time to treatment, output from these models is not included here. The regression models were solved using generalized estimating equations to account for the clustering of admissions within agencies.^*

Attendance, retention, and completion of treatment

Sample

Table 3 shows detailed breakdowns by service setting and year for each of the demographic, agency size, and performance measures. The sample included only adults who self-identified as having a primary problem of substance abuse. Affected others were therefore not included. The cross-sectional analysis uses adult individual service encounters as the unit of analysis. To attempt to control for possible organizational level characteristics that may have impacted any relative effect of PBC, the analytical data set was restricted to encounter-level data from those agencies that eventually became a part of the PBC system. Records for 2007 were pulled from agencies that became a part of the PBC system in 2008. Records for 2008 were pulled from PBC agencies. A total of five agencies provided only outpatient services. A total of 12 agencies provided both outpatient and intensive outpatient services. For encounters that occurred at the selected agencies, with a primary problem of substance abuse, the Maine data system included 2,528 OP encounters in 2007 and 2,631 OP encounters in 2008. In 2007 and 2008, 582 and 999 IOP encounters were recorded.

Table 3.

Treatment discharge data for PBC cohort agencies

	Outpatient		Intensive outpatient
	SFY07	SFY08	SFY07	SFY08
	(N=2,528)	(N=2,631)	(N=582)	(N=999)
	%	%	%	%
Age 18–25	31.4	29.1	28.7	30.5
White	93.9	94.4	95.9	95.4
Male	69.31	53.8a	59.1	51.7a
Married/cohabitating	26.3	22.8a	24.4	24.4
Primary drug: alcohol	57.5	52.0a	47.3	39.5a
Primary drug: marijuana	9.8	10.8	8.4	8.7
Primary drug: cocaine	6.5	9.2a	10.5	9.7
Primary drug: opiates	19.2	21.8a	23.7	29.2a
Co-occurring mental illness	21.4	52.3a	27.8	67.3a
Referred from CJ	31.5	25.1a	26.3	20.6a
Employed FT	34.8	28.0a	26.3	20.3a
No insurance	37.5	31.4a	26.6	21.8a
Large agency	70.5	73.3a	76.6	84.4a
Length of stay, 90 days+	40.3	23.9a	–	–
(PBC target)	–	(30.0)	–	–
Four or more sessions	69.4	55.1a	93.1	87.6a
(PBC target)	–	(50.0)	–	(85.0)
Completion of treatment	–	–	52.9	46.2a
(PBC target)	–	–	–	(50.0)

^a2007 vs 2008, p<0.05

Across both years and in both service settings, most of the service population was white and male. Alcohol was the most often cited primary drug of abuse. A substantial increase is visible in proportion of admissions noted as having a co-occurring mental illness from 1 year to the next, for both service settings, although this is likely due to the increased identification of persons with co-occurring mental illness in response to a statewide initiative designed to increase awareness of co-occurring conditions, rather than an actual change in the type of client population entering treatment.

Descriptions of agency-level variables and performance measures by level of care are also included in Table 3. As one would expect, approximately half of OP admissions, across both years, occurred at large agencies. Large agencies admitted most IOP admissions.

The percent of OP admissions staying 90 days or more decreased significantly from 40.3% to 23.9% from SFY07 to SFY08 (X²=160.318, df=1, p=0.000). Recall that, for OP services, agencies were required to have at least 30% of their admissions stay in treatment for 90 days or longer in order to meet PBC guidelines. PBC agencies that exceeded 40% would be eligible for an incentive payment of 1%. While the percent of admissions meeting the length of stay standard was near 40% in the year prior to the implementation of PBC, the percent fell quite dramatically in the year after implementation.

Similarly, the percent of OP admissions participating in at least four sessions decreased significantly from 69.4% to 55.1% (X²=155.237, df=1, p=0.000). Under PBC, for OP services, agencies were expected to have at least 50% of their admissions stay in treatment for four or more sessions. PBC agencies that had 65% or more of their admissions stay in treatment for four sessions or longer would receive incentive payments. Again, a downward trend is noted, although, as a whole, agencies appeared to be meeting the minimum standard.

The percent of IOP admissions staying four sessions or more decreased significantly from year to year, from 93.1% in SFY07 to 87.6% in SFY08 (X²=14.448, df=1, p=0.000). For IOP, agencies were required to have 85% of their admissions stay in treatment for four or more sessions. Agencies that had 90% or more of their admissions stay for four or more sessions were eligible for incentive payments. Once again, agencies fell below the incentive benchmark, when examined in aggregate.

Completion of IOP treatment also decreased significantly from 52.9% to 46.2% (X²=6.558, df=1, p=0.000). Whereas PBC required agencies to have a minimum of 50% of IOP admissions complete treatment, agencies during the year of PBC implementation did not meet the minimum standard on this measure. Clearly, most agencies did not meet the incentive level for this standard.

Regressions

The results of the logistic regression models are shown in Tables 4 and 5. The models provide an estimate of the overall strength and direction of the relationship between PBC and performance on client engagement within OP or IOP services, controlling for other relevant variables. Coefficients, standard errors, and significance levels are shown for all variables. The coefficient represents changes in the log odds of the focal variables of interest. Results are interpreted as odds ratios to assess relative effects. Based on these results, the effect of PBC on the probability of meeting treatment goals can then be discussed.

Table 4.

Summary table of OP logistic regressions

	OP 4 sessions		OP length of stay
	Model 1		Model 2
	(N=5,159)		(N=5,159)
Variables	Coeff (SE)	Exp(β)	Coeff (SE)	Exp(β)
Year=2008	−0.690* (0.216)	0.502	−0.774** (0.218)	0.460
PBC=yes	0.158 (0.230)	1.171	0.235 (0.150)	1.264
Year×PBC	−0.344 (0.259)	0.709	−0.175 (0.256)	0.840
Older adults	0.107 (0.099)	0.502	0.150 (0.096)	1.162
Male	−0.019 (0.087)	1.113	−0.087 (0.094)	0.917
White	0.000 (0.163)	1.001	0.066 (0.139)	1.068
Married/cohabitating	0.099 (0.067)	1.105	0.038 (0.051)	1.038
Mental illness	0.459** (0.136)	1.583	0.523** (0.125)	1.687
Criminal justice	0.219* (0.108)	1.245	0.208* (0.096)	1.231
No insurance	0.200* (0.091)	1.222	0.159 (0.087)	1.172
Employed full time	0.198* (0.078)	1.219	0.221* (0.076)	1.247
Opiates	−0.302* (0.141)	0.740	−0.219 (0.143)	0.803
Cocaine	−0.425* (0.153)	0.654	−0.411* (0.135)	1.024
Marijuana	0.025 (0.174)	1.025	0.045 (0.130)	1.046
Alcohol	0.056 (0.148)	1.062	0.024 (0.124)	0.663
Other drugs (ref group)	–	–	–	–
Agency size	−0.718* (0.227)	0.488	−0.622** (0.148)	0.537
Age and 2008	0.317* (0.139)	1.372	0.189 (0.110)	1.207
Gender and 2008	0.627 (0.328)	1.873	0.606 (0.338)	1.833
Mental illness and 2008	−0.668** (0.174)	0.513	−0.676** (0.158)	.509
Criminal justice and 2008	0.082 (0.144)	1.086	0.197 (0.131)	1.218
Intercept	1.004** (0.245)	2.730	0.751** (0.250)	2.120

^*p<0.05;

^**p<0.001

Table 5.

Summary table of IOP logistic regressions

	IOP 4 sessions		Completion
	(N=1,581)		(N=1,581)
Variables	Coeff (SE)	Exp(β)	Coeff (SE)	Exp(β)
Year=2008	−0.745 (0.641)	0.475	−0.478 (0.471)	0.620
PBC=yes	0.555 (0.399)	1.742	−0.101(0.230)	0.904
Year×PBC	0.330 (0.602)	1.391	0.231 (0.363)	1.260
Older adults	−0.051 (0.137)	0.951	0.194 (0.124)	1.215
Male	−0.101 (0.177)	0.904	−0.105 (0.093)	0.900
White	0.022 (0.276)	1.022	0.351** (0.101)	1.421
Married/cohabitating	0.271* (0.089)	1.311	0.099 (0.081)	1.104
Mental illness	0.547 (0.372)	1.728	−0.023 (0.184)	0.977
Criminal justice	0.327 (0.342)	1.387	−0.110 (0.203)	0.896
No insurance	0.156 (0.316)	1.169	−0.066 (0.128)	0.936
Employed full time	0.356** (0.055)	1.427	0.491 (0.102)	1.634
Opiates	−0.191 (0.181)	0.826	0.050 (0.130)	0.699
Cocaine	0.023 (0.180)	1.024	0.068 (0.229)	1.071
Marijuana	−0.014 (0.259)	0.986	0.161 (0.234)	1.174
Alcohol	0.210 (0.128)	1.234	0.241 (0.158)	1.273
Other drugs (ref group)	–	–	–	–
Agency size	−0.623 (0.347)	0.536	−0.239 (0.243)	0.787
Age and 2008	0.082 (0.201)	1.085	0.080 (0.214)	1.083
Gender and 2008	0.066 (0.215)	1.068	0.083 (0.113)	1.087
Mental illness and 2008	−0.707* (0.340)	0.493	−0.128 (0.227)	0.880
Criminal justice and 2008	−0.139 (0.388)	0.870	0.124 (0.288)	1.132
Intercept	2.199** (0.552)	9.018	−0.251 (0.284)	0.778

^*p<0.05;

^**p<0.001

In Table 4 (model 1), OP admissions that occurred during SFY08 were significantly less likely to stay for four sessions or more. The odds of OP admissions agencies staying for four sessions or more in SFY08 were 69% lower than the odds of SFY07 OP admissions staying for four or more sessions. This suggests that a temporal effect exists, rather than a PBC effect, as the estimates for PBC and the interaction term were not significant. Across both years of data, admissions of persons who had a mental illness, were referred by criminal justice sources or were employed full time were more likely to stay for four sessions or more. Admissions for a primary problem with cocaine and admissions to large agencies were significantly less likely to participate in at least four OP sessions, compared with admissions that had a primary problem with other drugs. During SFY08, admissions that were older were more likely to stay for four OP sessions, while admissions with a co-occurring mental illness were significantly less likely to stay for four OP sessions.

As shown in Table 4 (model 2), OP admissions among this group of agencies were significantly less likely to stay in treatment for 90 days or more during SFY08. PBC participation and the interaction of PBC and year were not significant predictors of length of stay. Admissions of persons who were referred from criminal justice sources, had a co-occurring mental illness, lacked health insurance, or were employed full time were significantly more likely to stay in OP treatment for 90 days or more. Males, admissions to large agencies, and those listing cocaine as their primary substance of abuse were less likely to remain in OP treatment for 90 days or more. During SFY08, admissions that were older or male were more likely to stay in treatment for 90 days, while admissions that had a co-occurring mental illness were less likely to stay in OP treatment.

Table 5 (model 1) demonstrates that IOP admissions that occurred during SFY08 were no more or less likely to stay for four sessions or more. In addition, participation in PBC did not have an effect. Across the two pooled years of data, admissions of persons who were employed full time were significantly more likely to stay for at least four IOP sessions or more, compared with their reference group.

For Table 5 (model 2), similar rates of treatment completion were found during both SFY07 and SFY08, and among both PBC and non-PBC agencies. Admissions of persons who were white or employed were significantly more likely to complete IOP treatment during the second year. No interaction effects were noted.

Discussion

As a whole, the results presented here suggest that PBC had only minimal effects on agency payments and no effect on time to assessment, time to treatment, level of client participation, length of stay, or completion of treatment.

The payment and utilization information presented here demonstrates that the financial and service delivery impacts of implementation were negligible. The mean payment adjustment was much less than 1% (0.01%), and the median payment adjustment approximated negative 1% (−1.13%). Overall, service delivery continued to be provided at acceptable levels, suggesting that agencies were not being selective in limiting admissions to only those persons who might have a greater likelihood of a successful treatment outcome. Given Maine’s past experience with PBC, this is a particularly important finding.

Did wait times for OP and IOP services within the PBC structure differ from wait times that occurred among ambulatory admissions that did not occur within the PBC structure? The results discussed here do not support the idea that access differed. PBC was not significant in any of the wait time analyses.

Recall, however, that agencies as a whole that were included within the TDS system were performing fairly well in terms of meeting the specified OP and IOP access measures. In particular, a high percentage of agencies met the standards for time between assessment and treatment, both among the PBC and the non-PBC agencies. Given such a high success rate, measurement of this construct may need to be revisited by the state agency to determine whether the desired length of time between assessment and treatment could realistically be shortened. In addition, the state should determine whether agencies are jointly performing assessment and treatment services, as the differentiation between these two services can be somewhat blurred in some service settings. At any rate, the PBC system is currently designed to strongly encourage agencies to shorten the amount of time between assessment and treatment. Further analysis into whether the achievement of this goal affects client engagement and outcomes will be of interest. In addition, as the PBC system in Maine evolves in coming years, further analyses can determine whether gains are made in improving the timeliness of access to treatment.

Results suggest that performance on level of participation, length of stay, and completion of treatment were significantly poorer the year after the implementation of PBC. Differences between years were more important than differences elicited by the implementation of PBC.

Several confounding factors may explain the decline in performance on these measures. A number of agencies were concurrently involved with a grant initiative that focused on using process improvement tools to increase access to and retention in outpatient and intensive outpatient treatment. While further analysis into the exact effect of participation in this grant initiative is warranted, a full examination is beyond the intent of this current article. One of the questions to consider, however, is whether agencies that initially focused their efforts on improving access may have inadvertently created systems challenges that stressed their organizational capacity to address issues of retention. Improving access might have increased the number of clients and the acuity of clients requesting services. The administrative data available do not allow for a close examination as to whether client-level characteristics shifted in a way that might have impacted these outcomes. Agencies that responded to increased admissions in a timely manner, by increasing staffing levels and by creating innovative ways to handle increased caseloads, may have performed better on the client outcomes discussed here compared with other agencies that were not as able to readily adapt.

The timing and size of the PBC incentives and penalties may have impacted overall results. Perhaps the quarterly timing was not frequent enough for agencies to be able to quickly identify opportunities for improvement. Increasing the frequency of payment adjustments may not be organizationally feasible from a state bureaucracy perspective, but changing the frequency of sharing program performance information with agencies may be both possible and beneficial. As agencies become more in tune with fluctuations in their performance, they may be able to develop and implement appropriate organizational improvements.

The small size of the financial consequences, as mentioned earlier, may limit the strength of the PBC system. Perhaps larger rewards or the possibility of more severe penalties would result in higher overall performance of agencies. The possible positive or negative impact of 9% of base funding may not have been significant enough to move agencies to adopt policies and procedures that would improve performance. The size of the incentives and penalties may also be relatively easily open to variation, as contracts are renewed from year to year. The state substance abuse agency will need to carefully consider whether adjustments to the incentive and penalty levels might be warranted.

One additional finding merits further discussion and research. Persons who were employed full time were significantly more likely to complete at least four OP sessions, stay in OP treatment for 90 days, and complete IOP treatment. While some prior research has examined the link between treatment outcomes and employment, most of this research has focused on whether participation in treatment improved post-treatment employment participation or earnings.^11–15 A more complete understanding of whether concurrent participation in employment and treatment leads to better joint outcomes could provide guidance for treatment agencies that are trying to implement work-friendly treatment programs.

Omitted variable bias may limit the utility of the overall analysis. Limitations on data collection do not allow for the detailed collection of clinical, diagnostic information about each admission, for example. Such information could have been useful in trying to understand whether client populations changed from year to year. Some of the additional information that is collected at the time of discharge may also warrant further exploration. As state-level data collection closely mirrors those indicators being tracked at the federal level as substance abuse services outcome measures, an examination of effects on abstinence at discharge, living arrangements, arrests, or employment status may also be of interest, although these indicators are not explicitly factored into the PBC system. In addition, selection bias may come into play as there are likely intrinsic differences between the PBC and non-PBC agency organizations. Future research will attempt to untangle these issues.

An additional issue to consider is whether adequate performance on agency-level variables actually translates into improved client outcomes. While other literature has linked agency-level performance on access and retention measures to improved client outcomes,^16–18 no such study has yet been conducted with the Maine treatment population. A more detailed examination of longer-term client outcomes may therefore be of interest. The current data system only captures data recorded at the last face-to-face session that occurred between a client and a treatment agency, however. Post-treatment data are not collected at the state level.

Beyond Maine, however, the value of the information presented here will be greatest to other state agencies that are interested in developing or evaluating PBC systems. The data are not only an important piece of an internal management feedback loop but also keeps individual providers informed about opportunities for improvement. As more and more entities within the substance abuse treatment field grapple with issues of how to ensure quality of care with limited resources, PBC may play an ever increasing role. While this article provides one example of how data from an effectively operational PBC system can be used to further systems improvements, additional studies are sure to follow.

Implications for Behavioral Health

The desire to improve accountability within behavioral health systems is leading many state behavioral health systems to consider the implementation of some version of PBC. Whereas bivariate results suggested that PBC and non-PBC agencies were performing differently in terms of times to assessment and treatment, logistic regressions suggest that the implementation of PBC in Maine did not have an effect on level of participation, length of stay, and completion of treatment among ambulatory services. The results described here are preliminary in nature and are presented to provide some baseline information about this new initiative.

The implementation of PBC did succeed in articulating and tracking performance measures. In future years, the State of Maine substance abuse agency will continue to track performance on these measures to determine whether gains are made. The actual value to the organization is more than what can be quantified here. By implementing PBC, the state agency was able to jointly improve accountability and reinforce the organizational focus on issues of access to and retention in treatment. Just implementing a PBC system that is both operationally effective and accepted within the substance abuse treatment community is an important measure of success. As the impact of PBC on specific performance measures is less clear, continued research should be conducted. As one of the first SSAs to implement a PBC, Maine can provide information to inform other state behavioral health systems as they move towards the implementation of similar structures.

The results found here should be tempered by the understanding that substantial additional analyses must be conducted to fully understand the impact of PBC on substance abuse treatment outcomes. With the framework of PBC solidly in place, Maine has the opportunity to revise measures, incentives, and penalties as contracts are developed each fiscal year. Slight changes to any one of these variables will provide an opportunity for Maine to revisit whether its PBC system is helping to reinforce an organizational emphasis on access and retention in treatment. As the Maine SSA digests these preliminary findings, the SSA must determine what policy changes, if any, might be warranted. Key systems level challenges facing Maine as it moves forward include expanding PBC to residential care and sharing the lessons learned with other agencies both within its larger department and within state government in general.