Housing and Employment Outcomes for Mental Health Self-Direction Participants

Abstract

Objective:

In self-direction, participants control individual budgets, allocating service dollars according to needs and preferences within program parameters to meet self-defined recovery goals. Mental health self-direction is associated with enhanced wellness and recovery outcomes at lower or similar cost than traditional service arrangements. This study compared outcomes of housing independence and employment between individuals who participated in self-direction and those who did not.

Methods:

This quasi-experimental study involved administrative data from 271 self-directing participants. Using coarsened exact matching with observed demographic, diagnostic, and other characteristics, the authors constructed a comparison group of non-self-directing individuals (n=1,099). The authors compared the likelihood of achieving positive outcomes between first and last assessments during the approximately 4-year study period for self-directing and non-self-directing individuals.

Results:

Self-directing participants were more likely than non-participants to increase days worked for pay or maintain days worked at 20 or more days in the past 30 days (number need to treat [NNT]=18; small effect size) and maintain or attain independent housing (NNT=16; small effect size), controlling, to the extent possible, for observed individual characteristics.

Conclusions:

Based on data from the nation’s largest and longest-standing program of its kind, mental health self-direction may be associated with modest improvements or maintenance of positive outcomes in employment and housing independence. This research adds to the literature examining self-direction in the context of mental health and begins to fill the need for a greater understanding of self-direction’s relationship to outcomes of interest to service users and families, providers, and system administrators.

Self-direction involves participants with serious mental health conditions controlling a portion of funds typically spent on their mental health treatment. Self-directing participants allocate individual budgets in a manner of their choosing within a set of program parameters, selecting and purchasing goods and services to work toward their goals. Typically, a coach or broker supports participants, facilitating development of person-centered plans, helping track progress toward goals and objectives, and assisting participants with financial management (1–3). Self-direction purchases are broad-ranging and may include clinical services and non-traditional expenditures such as transportation, eye and dental care, and computers (1, 4). Self-direction arrangements take different forms and are housed in and overseen by various entities, including mental health authorities, managed care, social service agencies, and advocacy organizations (5). At last count, approximately 700 individuals with serious mental health conditions participate in self-directed services in the United States (5), and mental health self-direction is expanding worldwide (6).

A 2014 systematic review examining 15 studies, including four conducted in the United States, concluded that although mental health self-direction is associated with increased quality of life and costeffectiveness, extant research had significant methodological limitations. The authors called for more rigorous research to inform policy and practice (7). Since that review, Spaulding-Givens and Lacasse (2) published a descriptive study of Florida Self-Directed Care (FloridaSDC). In that study, participants had modest improvements in mental health-related functioning and spent a majority of days in the community as opposed to institutional settings. A recent qualitative study documented self-directing participant gains in several domains, including vocational pursuits, independent housing, and community inclusion (1). Another study explored purchase types and concluded self-direction enables individuals to better address their needs through individualized strategies that support wellness and increase engagement in meaningful activities (4).

The present study adds to this evidence base. Its purpose is to compare two functional outcomes, employment and independent housing, between individuals who did and did not participate in mental health self-direction. These domains reflect self-direction’s value base, which is rooted in principles of recovery, independence, and community inclusion. A recent Institute of Medicine report highlighted similar indicators as key social determinants of health (9), and both were identified as priority outcomes in the President’s New Freedom Commission on Mental Health report (10) and the Substance Abuse and Mental Health Services Administration’s (11) Description of a Good and Modern Behavioral Health System. Service users, families, and administrators alike value these outcomes, which enhance quality of life and community inclusion and may reduce reliance on publicly funded systems. Using a quasi-experimental design with coarsened exact matching (8) and logistic regression, we estimated odds of change or maintenance of a positive outcome in employment and housing independence among individuals who were and were not self-directing.

This study uses data from FloridaSDC, the country’s oldest and largest mental health self-direction effort. There are two FloridaSDC programs in the state, which contracts mental health services through regional systems of care overseen by non-profit managing entities. These voluntary programs enroll approximately 330 individuals and are funded through a combination of state and local funds. To be eligible for either FloridaSDC site, individuals must have been at least 18 years of age and been designated as having a serious and persistent mental illness as determined by the state. Individuals must also have had permanent residence in the program areas and relied on public funds to cover mental health services. If individuals were not enrolled in Supplemental Security Income, Social Security Disability Income, or veterans’ benefits, they must have been in the process of applying for those benefits. FloridaSDC participants must have been legally competent to make financial decisions and may not have been enrolled in Assertive Community Treatment. SDC participants who were enrolled in public benefits received a yearly budget of approximately $1,900 per year, which could be used in addition to their existing insurance coverage. Uninsured individuals received an approximately $3,700 yearly budget, with half reserved for outpatient mental health treatment such as psychiatry and therapy. With support from a coach, and within program policy guidelines, participants linked purchases to specific recovery goals in their person-centered plans. Figure 1 depicts a breakdown of SDC spending by purchase category in one site (where such data were available). Most purchases were related to goods and services other than mental health treatment (some participants accessed mental health treatment using Medicaid or other benefits, and these services are not included in Figure 1), from transportation and dentistry to housing and employment-related supports.

Figure 1. Self-direction spending by purchase category for Program A, July 2010 to April 2015.

Notes: The full pre-match sample is included in the above figure. A sensitivity analysis was conducted with only the program participants who were successfully matched to a comparison case, and no major differences were found. Purchases included here depict only goods and services purchased using the self-directed budget, not all services used by SDC participants during the study period.

Methods

A university Institutional Review Board and the Florida Department of Children and Families’ Human Protections Administrator reviewed and approved the study protocol.

Data and Variables

Data were obtained as limited, de-identified datasets from the two managing entities that oversee the FloridaSDC programs, with consent from the state. According to state regulations, managing entities must collect demographics, service events, and mental health outcomes data. Providers report demographics to managing entities at admission and whenever that information changes, and mental health outcome assessments are reported at first assessment, quarterly thereafter, and discharge.

The datasets included demographics and outcomes information for all adults with a serious and persistent mental illness designation. Information regarding self-direction budget purchases and service utilization details were unavailable for both programs; therefore, this information was not analyzed in this study. The datasets cover different time periods because the managing entities assumed responsibility at different times and transferred the data at different times. The Program A study period spans 4.8 years beginning July 1, 2010. The study period for Program B covers three years, beginning July 1, 2012. Because of these disparate time periods, and because of demographic differences, the datasets were kept separate during the selection of matched comparison cases, and results are reported by site as well as in the aggregate.

Individuals enrolled in FloridaSDC were categorized as intervention group, and individuals who never enrolled in FloridaSDC were categorized as comparison group. Pre-cleaning, the dataset included 21,183 individuals, 403 of whom were SDC participants. Comparison group individuals who did not meet FloridaSDC eligibility criteria at any point during the study period (ever younger than 18, enrolled in Florida Assertive Community Treatment, or resided outside the geographic service area) were removed, as were individuals with only one assessment. After data cleaning and before matching, there were 403 SDC participants and 12,209 non-participants.

We created two binary dependent variables – employment and independent housing – by comparing measurements at first and last assessment for each individual. Outcome variables, selected based on study aims, were constructed to capture both positive change and maintenance of a positive outcome over the study period. There were three available indicators for exploring employment: employment status, income from paid employment (past 30 days) and days worked for pay (past 30 days). We selected the latter because it reflects work behavior (as opposed to income) most accurately and in greatest detail. Individuals with a positive employment outcome were those who either increased the number of days worked in the past 30 days at last assessment compared to first assessment or reported 20 or more days of work at both assessments. For housing independence, we constructed an outcome variable indicating transition from dependent housing (dependent living with others, group home, assisted living, supported housing, or hospital) or homelessness to living independently, or maintenance of independent housing status at first and last assessments.

We defined a positive program effect as either a positive change or maintenance of an already good outcome to account for the dual purposes of FloridaSDC; that is, helping participants gain access to employment and independent living, as well as preventing them from losing the resources they already have. The preventive goal is important to capture since the target population is at continued risk of losing access to employment or independent living. Only focusing on participants who lacked access to these resources at first assessment would miss the effects of the program in preventing loss of resources. For example, if SDC participants are more likely than non-SDC participants to remain living independently from first to last assessments, this positive outcome would not be evident from examining changes in independent living status over time.

Matching and Analysis

We used coarsened exact matching to create a non-SDC comparison group (8, 12). This straightforward method is designed to reduce imbalances between groups using specified categorical variables. The matching algorithm (available as an SPSS add-on) constructs a stratum for each combination of covariates and matches each intervention participant to one or more non-intervention individuals sharing the same stratum. The algorithm then produces weights to account for post-match group size differences (13). This method was chosen over other matching methods (e.g. propensity scores) because of the high number of categorical variables in the dataset and because it results in matched cases that share identical values (12).

Factors hypothesized to impact both the outcomes and participation into FloridaSDC were selected as matching variables (14). Variable choice was informed by reviewing pre-match descriptive statistics, key informant interviews with individuals knowledgeable about self-direction, and information gleaned from previous FloridaSDC evaluations (1, 2, 5, 15–17). We selected the following matching variables: age, high school completion, gender, ethnicity (Program A), race (Program B), schizophrenia diagnosis, substance use disorder diagnosis, marital status, county of residence, veteran status, limited English proficiency, arrests (ever arrested during the study period), activities of daily living (ever assessed as unable to perform activities of daily living during the study period), community tenure (ever spent one or more days out of the community during the study period), days between first and last assessments, and disability income receipt. We successfully matched 67% of the SDC participants (n=271) to 1,099 non-SDC participants. Post-match weights were used in all subsequent analyses.

We fitted two logistic regression models to estimate the likelihood of achieving a positive outcome on each of the dependent variables, controlling for relevant covariates. Model specifications were guided by a priori theory and methodological considerations. To control for first assessment differences in the outcomes, we included the first assessment value of the outcome measure as a covariate in each model. The set of standard covariates was: age, gender, race, ethnicity, high school completion, county of residence, program site, veteran status, marital status, physical disability, schizophrenia diagnosis, substance use disorder diagnosis, disability income receipt, days between first and last assessment, first-assessment value of the outcome, and any of the following during the study period: arrested, unable to perform activities of daily living, admitted to services as incompetent or involuntary, and spent one or more days out of the community.

Results

Descriptive statistics for the SDC and non-SDC groups before and after matching are presented in Table 1. Before matching, the groups were significantly different on nearly all measures. After matching, there were no statistically significant differences between the SDC and non-SDC groups.

Table 1.

Characteristics of SDC and non-SDC participants before and after matching

Variable	Before Matching					After Matching
	SDC (n=403)		Non-SDC (n=12309)		p	SDC (n=271)		Non-SDC (n=1099)		p
	n	%	n	%		n	%	n	%
Demographics and other characteristics
Age in years (M±SD)	52.15±10.61		44.86±14.46		.000	51.99±10.28		51.78±11.10		.769
Female	272	67	7593	62	.018	196	72	791	72	.908
White race	283	70	9541	78	.000	200	74	839	76	.381
Black race	94	23	1971	16	.000	56	21	215	20	.684
Hispanic ethnicity	20	5	1478	12	.000	8	3	28	3	.709
Completed high school at first assessment	319	79	7812	64	.000	222	82	899	82	.964
Married at any assessment	46	11	3134	25	.000	22	8	104	9	.493
Employed full- or part-time at first assessment	45	11	2121	17	.001	29	11	135	12	.469
Resides in county with the largest share of SDC participants	293	73	5170	42	.000	190	70	733	67	.283
Veteran of U.S. Armed Forces	43	11	488	4	.000	2	1	8	1	.986
English severely limited at any assessment	5	1	534	4	.002	0	0	0	0	-
Ever arrested	7	2	672	5	.001	2	1	10	1	.786
Diagnosis and assessments of disability and functioning
Ever diagnosed with a substance use disorder	78	19	3996	32	.000	42	15	201	18	.281
Ever diagnosed with schizophrenia	121	30	2644	21	.000	71	26	279	25	.784
Ever assessed as unable to perform activities of daily living	60	15	3500	28	.000	39	14	179	16	.445
Ever assessed as having a physical disability	59	15	1280	10	.006	39	14	152	14	.811
Receipt of public benefits and services
Days in system (M ± SD)	1045.15±401.18		881.52±498.87		.000	1025.14±404.38		1042.77±519.65		.546
Received psychiatric disability income at last assessment	297	74	3848	32	.000	197	73	803	73	.902
Ever spent one or more days out of the community	111	28	4151	34	.010	64	24	296	27	.266
Ever admitted to services as incompetent or involuntary	98	24	2788	23	.432	64	24	270	25	.744

Note: Tests for significance were two-tailed independent samples t-tests for continuous variables and two-tailed Pearson’s chi-square with asymptotic p-value reported for dichotomous variables. The dichotomous variables are coded 1 for clients who display the indicated t characteristic and 0 otherwise.

Table 2 presents unadjusted mean values of outcome variables at first and last assessment for SDC and non-SDC groups. The mean differences depicted in Table 2 do not capture maintaining a desirable outcome over the study period. To capture maintenance in addition to positive gains, we used the outcome variables described above rather than differences in means or proportions with improved outcomes for the multivariate analysis. Notably, SDC participants’ first assessments do not constitute a true baseline because 58% (n=158) of the SDC group were enrolled in FloridaSDC prior to the beginning of the study period. Some of the observed first-assessment differences may be due to having received FloridaSDC prior to the first observation. The regression models adjusted for this non-equivalence by estimating odds ratios (ORs) net of the first assessment values of outcome measures.

Table 2.

Unadjusted mean values of outcome variables at first and last assessment for SDC and non-SDC comparison group, by program site

Variable	SDC (Program A n=124, Program B n=147)					Non-SDC (Program A n=642, Program B n=457)
	First Assessment		Last Assessment		Unadjusted Difference between First and Last Assessment	First Assessment		Last Assessment		Unadjusted Difference between First and Last Assessment
	n	%	n	%		n	%	n	%
Program A
Days worked for pay in past 30 days (M±SD)	2.52±6.44		4.81±7.74		2.29	3.22±7.49		5.30±9.37		2.08
Lives independently alone or with others	107	86	112	90	4	448	72	506	81	9
Program B
Days worked for pay in past 30 days (M±SD)	1.33±3.95		2.14±5.76		.81	1.53±5.52		2.44±6.90		.91
Lives independently alone or with others	141	97	139	96	−1	328	73	361	81	7
Programs A and B
Days worked for pay in past 30 days (M±SD)	1.88±5.26		3.37±6.86		1.49	2.51±6.78		4.13±8.55		1.62
Lives independently alone or with others	248	92	252	93	1	785	72	875	81	8.9

Data were missing for the following variables in Program A: Days worked for pay in past 30 days for the non-SDC group (n=14) and lives independently alone or with others for the non-SDC group (n=16). Data were missing for the following variables in Program B: Lives independently alone or with others for the SDC group (n=2) and the non-SDC group (n=9).

We used logistic regression to produce adjusted ORs of SDC membership predicting the likelihood of a positive outcome, controlling for observed first assessment non-equivalences (Table 3). The logistic regression results indicate that, holding covariates constant, SDC participants in both programs were significantly more likely to experience a positive “days worked for pay” outcome compared to the non-SDC group (OR=1.73, p=<.01). SDC participants had more than twice the odds (OR=2.04, p<.05) of maintaining or attaining independent housing compared to the non-SDC group, controlling for covariates. We calculated two indicators of effect size: Cohen’s h and numbers needed to treat (NNT). Cohen’s h was .18 for employment and .21 for independent living. To achieve a positive employment outcome in one participant, 18 participants need to be enrolled in SDC for three years (the sample mean for length of enrollment). The corresponding number for a positive independent living outcome is 16 participants. As discussed earlier, these numbers reflect both improvements and maintenance of positive outcomes, and are adjusted for all model covariates, including outcome values at first assessment. They indicate small effect sizes, with a larger effect size on independent living than on employment.

Table 3.

Comparative change in outcomes from first to last assessment for SDC participants and comparison group, by SDC program and across both programs

Variable	Model N	OR	95% CI	df	p	−2 Log Likelihood	NNT	Cohen’s h
Program A
Increasing days worked in past 30 days, or maintaining days worked at 20 or more in the past 30 days	746	1.71	1.06–2.75	19	<.05	697.658	10	.23
Attaining or maintaining independent residential status	757	1.80	.87–3.74	19	.113	519.993	29	.14
Program B
Increasing days worked in past 30 days, or maintaining days worked at 20 or more in the past 30 days	604	1.86	.99–3.49	18	.054	316.331	21	.17
Attaining or maintaining independent residential status	598	2.96	1.09–8.03	18	<.05	301.094	15	.26
Programs A and B
Increasing days worked in past 30 days, or maintaining days worked at 20 or more in the past 30 days	1350	1.73	1.20–2.50	20	<.01	1087.627	18	.18
Attaining or maintaining independent residential status	1355	2.04	1.17–3.58	20	<.05	898.984	16	.21

Note: The study period was 7/1/2010 to 4/30/2015 for Program A and 7/1/2012 to 6/22/2015 for Program B. All models included a standard set of covariates: age, gender, race, ethnicity, high school completion, marital status, program site (for aggregated model only), county of residence, veteran status, physical disability, schizophrenia diagnosis, substance use disorder diagnosis, receipt of disability income, days between first and last assessment, ever arrested, ever assessed as unable to perform activities of daily living, ever admitted to services as incompetent or involuntary, ever spent one or more days out of the community, and the first assessment of the outcome.

Discussion

Using several years of administrative data, we found possible evidence of positive effects of selfdirection on both outcomes of interest: employment and independent housing.

Although these results are promising, findings should be regarded with caution due to study limitations. Although matching eliminated all observed differences between SDC and comparison groups, unobserved group differences likely remained. As is typical of quasi-experimental designs, matching and statistical controls minimized but did not guarantee the elimination of case-mix issues, so the results should be regarded as preliminary. Furthermore, only 67% of SDC participants were successfully matched, so approximately one-third of SDC participants were not included in the analyses. However, in preliminary analyses we used only ten variables to define the matching algorithm (versus the 16 variables in the final analysis); while approximately 90% of the sample was successfully matched, the balancing between the groups was inadequate. Therefore, we chose to sacrifice the number of matches for quality of matches, and the overall results were robust to these varying specifications. Future studies should collect more detailed background information so that a broader range of potential non-equivalence factors can be accounted for. It is, however, worth noting that complete inter-group equivalence is notoriously hard to establish in studies of socially complex outcomes, even with randomized control designs (18).

Second, participants’ first assessments do not constitute a true baseline because a majority of the FloridaSDC group were enrolled prior to the beginning of the study period. Reported group differences in outcomes between first and last assessments are net of first-assessment differences, possibly masking some intervention effects that might occur before first assessment. More accurate estimates of intervention effects would have been possible with a design utilizing true baseline data. To assess the degree to which the lack of a true baseline impacted our results, we repeated the analysis limiting the SDC group to the 158 participants who enrolled during the study period and thus, had a true baseline. For all models, the results were in a similar direction as they were in the full sample. SDC enrollment remained a statistically significant predictor of the employment outcome in the combined sample (OR=2.22, p<.01) while its effect on independent living did not reach significance (OR=1.87, p=.165). The difference in significance level is to be expected given the reduction in sample size, and the sensitivity analysis suggests the program effects reported above are not unduly biased by the lack of a true baseline.

Because the study was limited to available administrative data, potential confounders – including information about SDC-purchased goods and services and implementation-related factors such as informal screening practices – were not available in the data. Further, the data are collected by providers for billing and reporting purposes and are thus subject to possible unmeasured biases. While it would have been desirable to investigate self-direction’s relationship to service utilization, detailed service utilization information was not available.

Despite its limitations, this study has unique and important strengths. The outcomes have import for publicly funded mental health systems, and possible improvement in the areas of employment and independent housing suggests that persons are enhancing their independence and self-sufficiency. These findings are supported by a recent qualitative analysis of in-depth interviews with 30 FloridaSDC participants (1). In that study, ten interviewees reported gains in employment, five reported recent or planned transitions to independent housing, and nearly all reported pursuing valued roles in their communities. The present study’s quantitative findings, from a sample that includes nearly all eligible FloridaSDC participants, provides initial confirmatory evidence of findings reported by this smaller sample.

To the best of our knowledge, this study has the longest observation period of any mental health self-direction research. Self-direction is a complex intervention and requires a high level of participant engagement (1). Developing person-centered plans and working with a budget takes time. As such, gains may not be fully realized for months or years of participation. This study’s three-to-five-year observation period may provide a more representative picture of program effects compared to earlier studies.

Conclusions

This study adds to a small but growing body of literature addressing mental health self-direction’s effects. Compared to non-participants, self-directing participants were more likely to improve, or maintain at high levels, engagement in paid work and independent housing. We note, however, that there were marked case mix differences between groups, and there was uncertainty about whether the analytic approach adequately accounted for these differences. Future research should seek to corroborate these findings using more robust methods and might involve additional outcome domains such as service utilization and social connectedness, as well as alternative measures for employment, community integration, and independent living. Future work should involve mixed methods and implementation science approaches to explore issues associated with program reach (e.g. participant recruitment and formal and informal selection criteria) to further understand the specific mechanisms within self-direction that support recovery. For example, Croft and Parish (2016) found that participants’ relationships with their support brokers are critical for facilitating gains in self-esteem and independence, which were identified as core drivers in recovery. Studies establishing deeper and more nuanced understandings of program effectiveness could inform future research on program- and system-level factors. Future research should also examine cost implications of self-direction across health and social service systems; such information would support mental health leadership in expanding the use of self-direction to enhance self-sufficiency and quality of life.