Performance Assessment in Community Mental Health Care and At-Risk Populations

Abstract

We examine whether community mental health care centers (CMHCs) differ in their ability to serve at-risk populations, including clients with dual diagnoses for substance abuse, comorbid disabilities, and particularly severe functional impairment. Our analysis uses data from Indiana's public mental health system. Although at-risk clients experience, on average, worse outcomes than other clients, we find that some CMHCs achieve statistically significantly better outcomes than others. Although this information is useful to consumers and providers who wish to identify the most effective providers and treatment models for at-risk clients, it is not generated in standard performance assessments.

Introduction

While the health care system overall has retreated from managed care, such competition-based strategies continue to be increasingly prevalent in public mental health care (McIntyre, Rogers, and Heier, 2001). The organizational and financial arrangements associated with managed care are designed to increase efficiency and reduce waste in health care delivery. Concerns have arisen that these same incentives may lead providers to under-serve clients, particularly individuals with severe or complicated conditions (Ellis, 1998; Ware et al., 1996). To guard against such potential negative outcomes, managed care is typically supplemented with monitoring of provider performance. Yet, “[d]espite recent research on methods of risk adjustment…, the application of this research to Medicaid populations has lagged” (Ireys, Thornton, and McKay, 2002). For instance, standard methods of performance assessment focus on average outcomes, and may not detect suboptimal quality of care provided to select groups of at-risk clients.

Our analysis is based on data from the Indiana Division of Mental Health and Addictions (IDMHA). IDMHA is the public agency that serves as payer of last resort for persons with persistent and severe mental illness in Indiana. Care is delivered through 1 of 30 not-for-profit CMHCs, which act as gatekeepers to the 6 State hospitals. In 1996, the IDMHA adopted the Hoosier Assurance Plan that reformed the delivery system along managed care principles (Family and Social Services Administration, 1997). Subsequently, IDMHA produced provider report cards that describe various aspects of the centers that reflect the quality of care provided, including differences in assessed mental health outcomes experienced by clients at these centers (Family and Social Services Administration, 2000). While the IDMHA analysis controls for baseline functioning, it ignores variance that may be due to non-clinical client factors. In addition, the IDMHA analysis produces only limited subgroup analysis, in part because it uses a stratified approach that severely limits the extent to which different subgroups can be compared. As a result, the report cards cannot identify the vulnerability of some at-risk client groups, at-risk clients cannot use the information to identify optimal choices for people most like themselves, and treatments that work best on average may be applied to some clients for whom other treatment approaches may be more appropriate.

In previous analysis of these data (Deb, Holmes, and Deliberty, 2004), we showed the importance of adjusting performance measures for non-clinical client characteristics (e.g., sociodemographic variables and income), and different rates of client attrition across CMHCs. In this article, we extend this analysis to examine whether performance differentials observed in aggregate apply to specific, vulnerable sub-populations of clients, including clients with dual diagnoses for substance abuse, comorbid disabilities, and mental illnesses that cause particularly severe functional impairment.

Methods

Typically, estimates of provider performance have been generated in a fixed effects framework. We use instead a mixed random effects model to evaluate provider performance. The model includes both fixed coefficients (which permit control of client risk factors on outcomes) and random coefficients associated with provider-specific variation. We estimate the mixed random effects model in SAS^® with the PROC MIXED procedure (SAS Institute Inc., 1999). In addition, we adjust provider performance for different rates of client attrition using a non-linear selection equation. The formulation of the non-linear selection equation with fixed and random coefficients is described in Deb, Holmes, and Deliberty (2004) and estimated using the NLMIXED procedure in SAS^® (SAS Institute Inc., 1999).

As a multilevel modeling technique, the mixed random effects model offers a number of advantages over standard fixed effects specifications and is particularly attractive for the objectives of this analysis. First, because outcome analysis is typically based on data with a natural hierarchical structure (clients are grouped according to CMHC), multilevel models appropriately correct standard errors for clustering effects. Second, unlike analysis that is stratified by patient subgroups, mixed random effects models can accommodate the cell sizes that can arise when centers serve relatively small numbers of particular at-risk client types. Finally, multilevel models can easily accommodate interaction terms to evaluate if relative provider performance is conditional on type of client served (Goldstein and Spiegelhalter, 1996).

Data

The data for this analysis are taken from the Indiana Managed Care Provider Client Based Data Reports for fiscal years (FYs) 1998 and 1999. These data are collected on an ongoing basis for the universe of all clients who qualify for enrollment in the Hoosier Assurance Plan. An individual is eligible for care if (1) he or she has a severe mental, behavioral, or emotional disorder (as defined by the fourth revision of the Diagnostic and Statistical Manual of Mental Disorders) that is expected to last for more than 12 months, and that impairs functioning, (2) is at least 18 years of age, and (3) is eligible for Medicaid or food stamps, or has income that is below 200 percent of the Federal poverty level. The Indiana data include information on 35,098 individuals who were enrolled in the Hoosier Assurance Plan in FYs 1998 and 1999. Performance is assessed using the sample of 16,516 individuals who were enrolled continuously through this period and, thus, for whom we can observe changes in health status over 1998.

Dependent Variable

“High quality care for chronic conditions entails a focus on optimizing functional status” state Clauser and Bierman (2003). In this spirit, IDMHA requires CMHCs to routinely and comprehensively conduct functional assessment for every client for whom the division covers care, and bases its report cards on this information. Functioning is assessed using the Hoosier Assurance Plan Instrument - Adults (HAPI-A) (Family and Social Services Administration, 1997). The HAPI-A captures severity of illness on four behavioral health dimensions (symptoms of distress and mood, community functioning, social support, and risk behavior and substance abuse) and one dimension of physical health. The HAPI-A has been shown to yield reliable and valid measures of health outcomes for this population (Newman et al., 1997; Deliberty, Newman, and Ward, 2001). Centers are contractually obliged to report functioning scores biannually, and these data must be supplied before reimbursements are paid. Each center must have at least one designated person who receives training from IDMHA staff on an annual basis, where training is focused on achieving reliable scoring of clients. Reliability is further enhanced by an annual audit of a sample of HAPI-A scores undertaken by an accounting firm that uses trained medical personnel to evaluate the consistency with which functioning is assessed across clients and centers.

We base our analysis on one subscale from the HAPI-A, mental health symptoms and mood, which is constructed from ratings of the client's depression, anxiety, and symptom distress. The scale takes on values between 3 (most ill) and 21 (least ill). We chose to base our analysis on this one subscale because it is the one most highly correlated with the Global Assessment of Functioning Scale (a commonly employed measure of functioning used in the mental health care field), and because it has been shown to be more sensitive to changes in mental health, with effect sizes measured at 3-month intervals twice that of the Global Assessment of Functioning Scale (Newman et al., 1997).

Our measure of outcome is the change in the mental health symptoms and mood score between the beginning of FY 1998 and the beginning of FY 1999. Because the focus of our analysis is on persons with persistent severe mental illness, measured improvements on any outcome scale tend to be modest. Indeed, the average absolute change in our functioning measure is only 3.2 in this sample, and one-third of such changes were less than one in absolute value. Given that the instrument used to measure outcome in this analysis has been shown to have superior sensitivity to other instruments used in the field, we believe these minimal changes simply reflect the difficulty of achieving recovery in persons with such severe illnesses.

Case-Mix Variables

Given our choice of dependent variable, it is necessary to include baseline mental health functioning (as measured at the beginning of FY 1998) to control for possible effects of regression to the mean. We also consider a number of client socioeconomic characteristics to control for possible differences in illness perception, treatment efficacy, and compliance across different client groups. These include age and age squared (to account for possible non-linearities found in previous studies [Cuffel et al., 1996]), education level, sex, family income, marital status, and race and ethnicity. We also include two clinical variables, measured at baseline, including whether or not the client had a disability other than a mental disorder (including being blind, deaf, mute, non-ambulatory, neurologically impaired, developmentally disabled, or illiterate), and whether the center considered the client to be at risk for substance use. To correct for possible sample selection bias, we also include a second order approximation of the true selection index as recommended by Vella (1998). The selection index is derived from an analysis in which client retention across the 30 CMHCs is expressed as a function of observable client characteristics. Refer to Deb, Holmes, and Deliberty (2004) for more detailed information.

Center-Level Variables

Center-level effects are inferred from indicator variables in the mixed random effects model. To determine if CMHCs differ in their ability to serve particular at-risk populations, we interact these center indicator variables with risk indicators for client groups of concern. These groups include clients with dual diagnoses for substance abuse (ICD-9 codes 303, 304, and/or 305), clients with comorbid disabling conditions (including being blind, deaf, mute, non-ambulatory, illiterate, and/or having a developmental disability or neurological impairment), and clients with mental illness that causes severe functional impairment (as determined by an IDMHA algorithm that considers mental and physical health, social and community functioning, and risk behaviors). These interaction terms allow the slope coefficients on the client risk factors to vary by CMHC, and can be used to determine if a particular CMHC performs significantly better or worse when serving individuals from specific vulnerable client groups. Summary statistics for these variables are presented in Table 1.

Table 1. Prevalence of At-Risk Populations.

Sub-Group	Prevalence
	Percent
Mental Illness with Severe Functional Impairment	3.95
Co-occurring Substance Abuse	12.05
Comorbid Disability	28.67

NOTE: Results are based on all 35,098 clients served in FYs 1998 and 1999.

SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999.

Analysis Plan

We estimate two models, one with center indicators interacted with at-risk variables, and one without for comparison purposes. Overall model fit is evaluated using both Akaike and Bayesian Information Criteria (AIC and BIC). Both are related to the adjusted R² statistic, but with slightly different adjustments for the number of independent variables, and with smaller values of the test statistic associated with better model fit. Because the BIC also adjusts for sample size, it is less likely to favor over-fitted models.

To evaluate the robustness of center performance differentials across the at-risk client subgroups, we consider the following: First, the overall importance of the variation in performance across all CMHCs for different client subgroups is assessed by examining the covariance parameter estimates associated with each group of interaction terms. Standard t-statistics associated with the individual interaction coefficients can be used to determine the extent to which any one CMHC may produce statistically significantly better or worse outcomes for a particular at-risk client subgroup. Second, we calculate the largest change, both positive and negative, in ranks inferred from the center indicators in the overall model and the coefficients on the interaction terms associated with each at-risk client group. Third, we calculate the proportion of changes in relative ranks between each at-risk group and the overall center ranks. The number of changes in relative ranks is given by 1-W/2, where W is Kendall's measure of concordance between the ranks implied by the two groups being compared. Fourth, we calculate the correlation in relative performance and implied ranks between each at-risk group and the overall client population.

Results

Model fit statistics are presented in Table 2. Based on both selection criteria, the model that includes interaction terms dominates the model that assumes relative performance differentials are the same for all client subgroups. The covariance parameter estimates, also presented in Table 2, indicate that there are significant differences in relative center performance for each at-risk group, with the most significant differences being observed for the client group with severe functional impairment due to mental illness (p=0.012), followed by the group with co-occurring substance abuse (p=0.022).

Table 2. Model Fit and Covariance Parameters.

Category	Overall Model Fit		Covariance Parameter
Model	AIC	BIC
Base Model	88750.6	88753.4	—
Intercept	—	—	0.2382 (0.0004)
Interaction Model	*88686.7	*88693.7
Intercept	—	—	0.2363 (0.0007)
Mental Illness with Severe Functional Impairment	—	—	1.1781 (0.0118)
Co-occurring Substance Abuse	—	—	0.3473 (0.0221)
Comorbid Disability	—	—	0.1502 (0.0290)

^*Indicates best model based on fit criteria.

NOTES: N=16,516. AIC is Akaike Information Criteria. BIC is Bayesian Information Criteria. Numbers in parentheses are p-values.

SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999.

Estimates of the fixed coefficients that capture the effect of client case-mix variables (available from the authors on request) are robust to the inclusion of interaction terms. The random effects solutions in Table 3 provide information on the relative performance of the 30 community mental health centers. These random effects coefficients represent the estimated deviations for each center from the mean performance score, with positive (negative) estimates indicating the center performed above (below) the average level. The coefficients on the interaction terms, by comparison, represent the estimated deviations in performance score between the at-risk group considered and the not at-risk group for that particular CMHC.

Table 3. Random Effects Solutions for Provider Performance Differentials.

			Interaction Model
	Base Model		Intercept (Not At-Risk)	Implied Rank	Mental Illness with Severe Functional Impairment	Implied Rank	Co-occurring Substance Abuse	Implied Rank	Comorbid Disability	Implied Rank
CMHC	Intercept	Implied Rank
A	*1.1144	1	*1.2174	1	-0.0647	3	**-0.4342	3	-0.2577	1
B	*0.7459	2	*0.7375	2	0.9782	1	0.2031	1	-0.1537	4
C	*0.6995	3	*0.7048	3	0.0214	7	0.063	4	NA	—
D	*0.4767	4	*0.5459	4	0.6069	2	*-0.6715	13	0.3127	2
E	**0.4011	5	0.2679	5	-0.977	23	0.0488	8	**0.5370	3
F	**0.2751	6	0.1917	7	0.6472	5	0.0834	9	0.3918	5
G	0.253	7	0.178	10	0.286	9	0.3794	5	0.0106	11
H	0.1783	8	0.1807	9	0.9	4	-0.4708	17	0.0081	10
I	0.1763	9	0.0159	14	0.558	8	NA	—	0.2405	7
J	0.148	10	0.2134	6	-0.0222	11	0.2845	6	-0.2852	15
K	0.1233	11	0.1831	8	-0.244	16	-0.3539	14	NA	—
L	0.1147	12	0.147	11	0.65	6	-0.2181	12	-0.1091	13
M	0.1127	13	0.1003	13	*-1.944	29	*-0.7003	20	**0.3756	6
N	0.0632	14	0.1467	12	-0.0001	13	0.2302	7	*-0.6207	21
O	0.0528	15	-0.0129	16	0.0013	15	-0.1696	15	0.2083	9
P	0.0223	16	-0.083	18	-0.7651	24	*0.9786	2	-0.3302	20
Q	-0.0092	17	-0.1926	22	-1.1396	28	-0.0239	16	0.4182	8
R	-0.0191	18	0.0127	15	-0.3434	21	-0.0551	11	-0.1921	16
S	-0.0374	19	-0.0942	19	0.0832	14	0.1055	10	0.077	14
T	-0.0382	20	-0.1956	23	-0.0369	19	NA	—	0.3198	12
U	-0.1355	21	-0.06	17	**-0.8093	25	NA	—	-0.2875	19
V	-0.2475	22	-0.1518	21	0.0164	18	**-0.8166	23	NA	—
W	-0.2524	23	-0.1301	20	-0.3294	22	*-0.8571	24	-0.1792	18
X	**-0.3646	24	*-0.5014	26	0.9018	10	0.032	19	0.2013	17
Y	*-0.3793	25	**-0.3481	24	-0.6612	27	-0.0916	18	-0.2189	24
Z	*-0.4412	26	**-0.4256	25	0.3262	17	NA	—	-0.1033	22
AA	*-0.5352	27	*-0.5064	27	-0.4176	26	-0.1367	21	-0.0477	23
BB	*-0.5923	28	*-0.5581	28	0.7308	12	-0.31	22	-0.0865	25
CC	*-0.7936	29	**-0.5870	29	0.2699	20	NA	—	-0.2682	26
DD	*-1.112	30	*-0.9962	30	*-2.295	30	NA	—	0.0393	27

^*p<0.05.

^**p<0.10.

NOTES: CMHC is community mental health care center. NA indicates the CMHC served no clients in the at-risk category. At-risk coefficients under the interaction model represent the marginal differences in performance, while at-risk ranks are based on levels of performance.

SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999.

In the overall model, four CMHCs are found to perform significantly (p<0.05) better than average, and six perform significantly worse than average. Although the magnitude of these provider-level coefficients may appear to be small, it is important to note that they measure the deviations from the average change in functioning score. Given the mean absolute change in functioning score is only 3.2, even a 1-point difference would be considered substantial. Also, the coefficients measure the average deviation for all clients treated at the center. Thus, a coefficient of +1 would correspond with improving the functioning of every client at the center by one additional point (on average) compared with the mean center. Improvements above +1 are in the top one-third of all improvements for this population, so a center with a coefficient greater than one would have essentially moved their clients from outcomes in the middle one-third of the distribution to outcomes in the top one-third of the distribution.

Across all centers, outcomes are much worse for clients whose mental illnesses caused severe functional impairment (average interaction coefficient of -0.10), and somewhat worse for clients with co-occurring substance abuse (-0.027) and comorbid disabling conditions (-0.027). Although there are only a small number of significant differences for the at-risk groups, at least one CMHC performs statistically significantly worse for every at-risk group considered. In addition, the variability in performance is much greater for clients whose mental illnesses cause severe functional impairment than for other clients.

A comparison of the implied ranks across the subgroups considered (also provided in Table 3) reveals the largest change is between the ranks for the overall client population and the ranks for the group with mental illness causing severe functional impairment. The maximum changes were a 53-percentile increase and a 57-percentile decrease in rankings), followed by the group with co-occurring substance abuse (with a 54-percentile increase and 38-percentile decrease, respectively), followed by the group with a disabling comorbidity (with a 26-percentile increase and a 33-percentile decrease, respectively), and lastly followed by the not at-risk group (with a 13-percentile increase and a 17-percentile decrease, respectively). This ordering is preserved when comparing the proportion of relative ranks that change between the overall and at-risk rankings: rank reversals are nearly nine times more likely between the overall ranks and the ranks for the group with mental illness causing severe functional impairment than between the overall ranks and the ranks for the not at-risk group. Rank reversals for the groups with co-occurring substance abuse and other disabling conditions are, respectively, seven and three times more likely than for the not at-risk group.

Correlation coefficients are presented in Table 4. The correlations across estimated performance differentials are statistically insignificant and only weakly positive in size for all at-risk groups considered and the overall client population. In contrast, the correlation between the overall differentials and the not at-risk differentials is 0.98 and highly statistically significant (p=0.000). Thus, it appears that relative center performance overall is determined largely by its ability to serve less vulnerable clients. Although correlations between implied ranks are, by contrast, statistically significant, the strength of association is only moderately strong, particularly between the overall ranks and the ranks for the group with mental illness causing severe functional impairment.

Table 4. Correlations in Performance Differentials Across Patient Subgroups.

Category	Not At-Risk	Mental Illness with Severe Functional Impairment	Co-occurring Substance Abuse	Comorbid Disability
Overall	*0.977/0.968	*0.286/0.653	*0.089/0.719	*0.130/0.889
Not At-Risk	—	*0.282/0.626	*-0.021/0.718	*-0.012/0.811
Mental Illness with Functional Impairment	—	—	*0.025/0.443	*-0.106/0.500
Co-occurring Substance Abuse	—	—	—	**-0.271/0.395

^*p<0.05.

^**p<0.10.

NOTES: Pearson's correlation of performance differentials/Spearman's correlation of ranks.

SOURCE: Holmes, A.M., Indiana University-Purdue University Indianapolis and Deb, P., Hunter College, City University of New York: Data based on estimates from the Indiana Managed Care Provider Client Based Data Reports, FYs 1998-1999.

Discussion

The President's New Freedom Commission on Mental Health (2003) identified outcome assessment and accountability as unique challenges to the successful functioning of the mental health care system. Problems of asymmetry of information, in which providers know more about patients' conditions than either insurers or patients themselves, are particularly acute in mental health care and, combined with incentives for risk selection, can place the neediest patients in peril (Frank and McGuire, 2000). Outcome assessment is needed to ensure these quality problems are not exacerbated by managed care delivery systems that increasingly characterize publicly funded community mental health care.

Access to community-based care for persons with even the most debilitating mental illnesses was advocated by the New Freedom Commission on Mental Health (2003) which recognized that mental health care should be consumer and family driven. With this authority comes the responsibility for selecting optimal care from community providers and the need for policymakers to provide the information consumers need to make these choices, including those consumers with particularly severe or complicated conditions.

The Commission also reported that disparities exist in access to appropriate mental health care and the burden of mental illness borne by certain segments of the population. In particular, the care for persons with co-occurring disorders was found to be inadequate. Administrators of public mental health care systems need to consider the extent to which they meet the needs of such at-risk subgroups. Similarly, researchers who undertake effectiveness research to identify best treatment practices need to consider not only what works best for the typical client, but also whether these same practices are optimal for more vulnerable clients.

Standard provider-profiling exercises fail to identify whether some providers are particularly effective in the treatment of the most vulnerable at-risk clients, and these clients cannot use the resulting information to identify optimal choices for people most like themselves (Elliott et al., 2001). While stratified analysis has been suggested as a possible solution to these problems, strata-specific risk rates typically have unsatisfactory statistical properties, particularly for under-represented client groups (Gatsonis et al., 1995). This feature is particularly undesirable if the most vulnerable at-risk clients are infrequently encountered in CMHCs.

In this article, we used a mixed random effects model to evaluate provider performance. Compared with standard provider-profiling exercises, such models yield more precise estimates of relative performance, especially when sample sizes are small. In addition, the model easily accommodates interaction terms to evaluate whether performance differentials are robust across various client subgroups. Our results suggest that, for some CMHCs, relative performance is significantly dependent on the type of client served; while, on average, centers attained poorer outcomes for at-risk clients than less vulnerable clients, the discrepancy was larger for some centers than others. Furthermore, the estimated performance differences for at-risk populations were only moderately related to overall performance differences, with the result that standard provider profiles sometimes failed to identify the most effective providers of care for at-risk clients. We also found that performance differentials varied much more for clients with mental illnesses that resulted in severe functional impairment than for clients with less severe illnesses. Policymakers need to be aware that in such situations these at-risk clients may have to travel relatively greater distances to obtain quality care, further aggravating disparities in health status and access to health care (Dranove et al., 2003).

While the number of centers with statistically significantly better (or worse) outcomes for various at-risk client groups may be small, the results still have practical relevance. By identifying a small number of exemplary centers, we have identified centers whose practices, etc., may be worth emulating by other providers. Similarly, by identifying a small number of centers with subpar performance, we have identified centers where quality improvement initiatives by State agencies could be most effectively applied. Our results can also be used to assess the distribution of quality care across different regions of the State, both overall and with respect to vulnerable subgroups.

Although we believe our empirical model offers a number of advantages over standard specifications, a number of caveats deserve mention. First, the analysis is based on only one clinical measure—change in mental health symptoms and mood over a 1-year period. Our relative rankings may discriminate against centers that place more priority on other dimensions of mental health (e.g., community functioning, reduction of substance abuse risk), or that focus on longer or shorter time horizons. Second, our results are based on data for a single State over a single year. The external validity of our findings may be limited to the extent that system, practice, or client differences may exist across geographical regions or time, although the methods we have presented for detecting differences in provider performance for vulnerable at-risk populations remain valid regardless of setting. Third, our analysis can only consider differences in performance across CMHCs, and not differences within a given CMHC. As a consequence, our results cannot inform consumers and insurers about the relative effectiveness of individual providers or treatments. However, given that IDMHA clients must select annually a CMHC to serve as a mental health care gatekeeper (rather than a specific provider or treatment protocol), center-level comparisons remain useful. Finally, our results only indicate that differences exist, not why they exist. One of the advantages of the methods used in this article is that it is possible, in theory, to incorporate center-level variables in the mixed random effects specification to identify center characteristics associated with better performance. Empirically, however, our ability to assess the impact of multiple center-level characteristics is limited given the small number of centers on which our analysis is based. The results of this article do provide a critical first step in quality improvement—having identified exceptional centers, policymakers can use this information in future studies to help determine the staffing, practice patterns or organizational structures that are associated with superior or inferior outcomes.