Randomized Trial of Supported Employment Integrated with Assertive Community Treatment for Rural Adults with Severe Mental Illness

Abstract

Urban-based randomized clinical trials of integrated supported employment (SE) and mental health services in the U.S. on average have doubled employment rates of adults with severe mental illness (SMI) compared to traditional vocational rehabilitation. However, studies have not yet explored if service integrative functions of SE will be effective in coordinating rural-based services that are limited, loosely-linked, and geographically-dispersed. In addition, SE’s ability to replicate work outcomes of urban programs in rural economies with scarce and less diverse job opportunities remains unknown. In a rural South Carolina county, we designed and implemented a program blending assertive community treatment (ACT) with an SE model, Individual Placement and Support (IPS). The ACT-IPS program operated with ACT and IPS subteams that tightly integrated vocational with mental health services within its self-contained team. In a 24-month randomized clinical trial, we compared ACT-IPS to a traditional program providing parallel vocational and mental health services, on competitive work outcomes for adults with SMI (N = 143, 69% schizophrenia, 77% African American). More ACT-IPS participants held competitive jobs (64% vs. 26%, P < .001, ES = 0.38) and earned more income (Mdn = $549, interquartile range (IQR) = $0 – $5145 vs. Mdn = $0, IQR = $0 – $40, P < .001, ES = 0.70) than comparison participants. Competitive work outcomes of this rural ACT-IPS program closely resemble those of urban SE programs. However, achieving economic self-sufficiently and developing careers probably requires increasing access to higher education and jobs imparting marketable technical skills.

In the U.S., point prevalence of competitive employment for adults with severe mental illness (SMI), work impairments, and limited job experience ranges between 10 and 20 percent.^{1, 2} Fourteen U.S.-based randomized clinical trials of supported employment (SE) have on average doubled these rates. Meta-analysis of the first six trials reported relative equivalence across SE programs on multiple competitive work outcomes (e.g., employment rates, weeks worked, and income earned), and a consistent two- to three-fold improvement over traditional vocational rehabilitation.³ The second eight trials, constituting the federally-sponsored “Employment Intervention Demonstration Program,” replicated these work outcomes.⁴ Based on cumulative findings of SE program effectiveness, federal agencies and private foundations have recently funded SE adoption efforts throughout U.S. public mental health systems.⁵

Most SE trials have taken place in large U.S. urban areas. Verifying SE services as effective in diverse rural settings requires additional studies addressing at least two crucial questions. First, can urban SE program strategies that coordinate vocational with specialty mental health services^{6, 7} be effectively implemented into under-resourced, and fragmented rural service arrays? Second, and more importantly, will rural-based SE programs achieve similar work outcomes as their urban counterparts in southeastern U.S. rural economies, characterized by limited job opportunities, high unemployment, and concentrations of severely disadvantaged African American residents?

Service coordination

Although rural service profiles vary enormously across U.S. regions,⁸ implementing SE programs’ coordinating functions into loosely arrayed, weakly linked, understaffed, and geographically-dispersed rural services mixes is extremely difficult.^{9, 10} In rural South Carolina, the site of our study, obstacles to integrating SE with mental health services may be more challenging to overcome compared to other U.S. rural regions. First, South Carolina’s government operates public services with a very small budget relative to most other states. Second, because a much higher proportion of the state’s population resides in rural areas (40%) compared to the U.S. as a whole (21%), the state faces overwhelming challenges to staff rural-based health and social services for persons scattered over large geographic expanses.

Third, African-Americans comprise a very high proportion of the state and study area’s population (30% and 47%, respectively). They experience a disproportionately high prevalence of multigenerational poverty and chronic medical illness compared to other racial/ethnic groups.^{11, 12} Even worse, African-American adults with SMI use mental health services much less than other racial/ethnic groups.¹³ Their low service use and large rural presence has seriously interfered with South Carolina’s efforts to develop accessible vocational, mental health, and other services responsive to their needs and preferences. Insufficient psychiatric treatment combined with harsh social and economic disadvantage, here and in other southeastern rural regions, probably accounts for African Americans’ poorer psychiatric status and functioning compared to other racial/ethnic groups.^{12, 14} Thus, coordination of rural SE and mental health services may not only considerably improve their accessibility, but may also induce synergistic therapeutic effects that alleviate illness and promote labor market success.

Employment outlook

Job opportunities in southeastern rural areas are limited for adults in general,¹² and especially for those with SMI. First, poorly-resourced secondary schools grapple with graduating their students. More than one-third of rural African-American and one-quarter of rural White students do not complete high school.¹⁵ During their high school years, students who experience onset of prodromal and/or full psychiatric syndromes, or who exhibit risk factors for future illness, may be less likely to graduate. However, few studies have examined associations between illness and graduation rates.¹⁶ Second, for those who earn diplomas, prospects for obtaining any further education and jobs offering advanced skills training are grim. Cumulative impacts of low educational attainment, technical expertise, employment experience, and job availability severely handicaps adults with SMI in vying for attractive jobs, while markedly increasing their risks of remaining mired in unemployment and poverty.¹⁷ Thus, focusing integrated SE and mental health services toward employment in rural labor markets may help extract the most out of meager job opportunities and services .

To assist rural adults with SMI in accessing and benefiting from integrated vocational and mental health services oriented towards increasing chances of competitive labor market success, the South Carolina Department of Mental Health (SCDMH) and the Medical University of South Carolina (MUSC) collaborated to conduct one of the eight site studies of the Employment Intervention Demonstration Program (EIDP).¹⁸ Each site selected and implemented its own SE and comparison programs and evaluated them with the EIDP’s comprehensive cross-site assessment protocol. For the Sumter County catchment area of the SCDMH-operated Santee-Wateree Community Mental Health Center (SWCMHC), we designed and attempted to implement two empirically-supported urban SE service models, assertive community treatment with integrated vocational rehabilitation (ACT-IVR)¹⁹ and Individual Placement and Support (IPS).²⁰ ACT-IVR tightly integrates vocational with mental health services within a self-contained provider team. IPS integrates its vocational services with its host agency’s or another agency’s mental health services.^{7, 21, 22}

Our comparison program, a formal partnership between a local vocational rehabilitation agency and SWCMHC, provided parallel traditional vocational and mental health services, respectively. The rehabilitation agency’s employment specialists prepared participants for competitive jobs, by taking a gradual, stepwise approach that focused on developing skills to manage workplace demands in staff-supervised jobs, set-aside for adults with work disabilities.

In a 24-month randomized clinical trial, we addressed five specific questions: (1) What obstacles would a rural South Carolina setting pose to implementing and sustaining two evidence-based SE models (ACT-IVR and IPS) that integrate SE with specialty mental health services? (2) How do competitive work outcomes of the ACT-IVR and IPS programs compare to a traditional program providing parallel services in a rural economy with a high proportion of extremely disadvantaged African-American residents? (3) How are overall work outcomes affected by differing strategies for obtaining employment: (a) quickly pursuing work in the open labor market (ACT-IVR and IPS) versus (b) eliminating employment barriers by immediate placement into temporary set aside jobs (traditional program)? (4) How do rates of income earned from competitive work change over time by program, a potential proxy for recovery of work capacities and community integration? (5) Secondarily, how do the programs’ work outcomes compare after controlling for the number of participants’ service contacts, and accounting for potential moderating effects of six pre-selected characteristics (job history, benefit status, diagnosis, education, race, age, and gender) commonly found in prior studies to be associated with work outcomes?²³

Methods

Geographic and Service Setting

From its main office in the Town of Sumter (population 39,000), SWCMHC serves four of the state’s 46 counties, including the Sumter County study service area (population = 102,000; 665 square miles; density = 157/square mile). African American (47%) and White (49%) residents, neither of Hispanic/Latino origin, make up almost the entire population. In 1999, median county per capita income hovered near $16,000; 14% of persons aged 18 and older, and 13% of families, lived below the federal poverty level. About 62% of persons were in the labor force (56% for women); unemployment rates ranged from a high of 5.5% in 1997 to a low of 4.6% in 1998. Jobs clustered in economic sectors of retail trade (12%); manufacturing (24%); and educational, health, and social services (19%).

Selection Criteria, Recruitment, Enrollment, and Random Assignment

(1.4 and 2.8) The EIDP Steering Committee defined the population for this study as meeting both the federal Center for Mental Health Services’ criteria for severe and persistent mental illness, based upon diagnosis, illness duration, and level of disability,²⁴ and EIDP-specific criteria of age 18 or older; unemployed at study entry; and current and/or future interest in competitive employment.⁴ For our South Carolina site-study, we added the criterion, “SWCMHC client for at least six months,” to reduce the probability of excessive attrition due to frequent relocation of residence typical of this rural south population. We intended these selection criteria to produce a sample representative of a broader population of adults with SMI and work disabilities, who were not currently in crisis and who expressed interested in working, versus a sample adults with the most severe illnesses and disabilities typically enrolled by “Madison-model”ACT programs. The full-convened institutional review boards of both SCDMH and MUSC approved the study plan in 1995 and amendments throughout the project period.

In 1996, SWCMHC estimated 1,600 to 1,800 of its 2,800 Sumter County Community Support Program (CSP) working-age clients would likely meet eligibility criteria. (2.9) We asked SWCMHC providers to notify their clients of this study during routine service encounters. However, we do not know how many clients actually learned of the study during the 24-month recruitment window. SWCMHC providers furnished us with written documentation that 285 clients definitely requested study details. Using a brief screener of eligibility criteria, providers determined 20 as ineligible, and judged another 37 as probably eligible, but not interested. We later verified that all of these provider judgments of client eligibility were accurate. Research assistants (RA) met individually with the remaining interested clients (n = 228) to obtain voluntary informed consent, describe study purposes and procedures, and reassess eligibility. All RAs were licensed or credentialed service providers, or persons who had worked with the study population for at least 15 years. They determined 17 clients as ineligible; another 21 withdrew consent after their meetings. Remaining clients (n = 190) attended one RA-led research induction group meeting to further review study activities and address clients’ concerns. After this meeting, 13 withdrew consent, yielding a final pool of 177 enrolled participants.

The random assignment protocol was generated and implemented by two MUSC research personnel otherwise unaffiliated with this study. A statistician prepared a SAS-generated restricted random assignment sequence (permuted blocks of three) to balance time of study entry and sample size across study programs. To a Charleston-based RA, he surrendered the assignment sequence, to which no investigator was permitted access. Sumter RAs transmitted enrolled participants’ names to the Charleston RA, who assigned participants’ to programs.

Implementation Difficulties and Study Redesign

Our original study plan called for a three-group trial comparing two newly-implemented SE programs, ACT-IVR^{19, 25} and IPS,²⁰ to a traditional vocational rehabilitation program. We contracted with developers of the ACT-IVR and IPS models who, as independent external consultants, would train and mentor program staff, and monitor program fidelity to ACT and IPS model standards throughout the 4-year project period.

Two months into the project period, SWCMHC had not yet recruited the requisite qualified providers to staff both the ACT-IVR and IPS programs, similar to recruitment problems described by other rural ACT programs.²⁶ The ACT-IVR and IPS consultants advised the investigators that simultaneous replication of these two model programs would not rapidly be achieved, and recommended modifying the study plan. At this point, 7 participants had been assigned to each program. Between months 2 and 6 of the project period, the CMHS program official, the EIDP Steering Committee, the external consultants, and the investigator team explored alternative study designs, while slowing participant accrual to the ACT-IVR and IPS programs to preserve low staff/participant ratios as stipulated by their respective model standards. During this deliberation period, 20, 27, and 31 participants were assigned to the ACT-IVR, IPS, and traditional programs, respectively, for a total of 27, 34, and 38.

We integrated the partially implemented and incompletely staffed ACT-IVR and IPS programs into a single ACT-IPS program, operating with ACT and IPS subteams composed of the ACT-IVR and IPS staff members, respectively. We eliminated the IPS condition and released its 34 participants from the project. The 27 ACT-IVR participants remained enrolled in the ACT-IPS program; the 38 traditional program participants continued without interruption. We assigned the subsequent 78 enrollees as pairs to the ACT-IPS and traditional programs, using the first two of the three elements in the random assignment sequence, giving a final participant tally of 66 (39 + 27) and 77 (39 + 38), respectively.

Sensitivity Analyses Assessing Threats of Internal Validity

Project redesign may have disrupted experiences for the 27 participants who began in the ACT-IVR program but transferred to the ACT-IPS program. Additionally, study experiences of these 27 participants may have been discordant with those of the 39 subsequently assigned directly to the ACT-IPS program. To explore potential confounding induced by redesign, we specified a series of sensitivity analyses (available from the authors), comparing program and participant activity before and after redesign. The ACT-IPS program provided a modestly higher proportion of vocational services compared to the ACT-IVR program, but no substantive differences between ACT-IVR and ACT-IPS participants emerged across personal characteristics; types and amounts of services received; attrition rates; and work and other outcomes. However, such minor differences do not rule out that other potent but unobserved factors associated with redesign account for observed variation in work outcomes.

Program Operations after Study Redesign

ACT-IPS program

The ACT subteam consisted of a staff profile typical of rural ACT teams:²⁷ one psychiatrist, one MA-level clinician, two BA-level registered nurses, and two BA-level case managers. The IPS subteam consisted of an MA-level social worker, who led the ACT-IPS program, and two BA-level employment specialists. An administrative assistant handled logistics for the entire program. About a third of the staff was African American and the remainder White. To tightly integrate vocational and mental health services, the subteams met daily together as a full program to allocate tasks to each IPS specialist and ACT staff member; formulate treatment plans emphasizing employment goals according to each participant’s preferences, strengths, and weaknesses; select a mix of services tailored for each participant’s work goals; and update each participant’s single unified treatment record.

The ACT subteam operated as a “generalist” (i.e. interdisciplinary) team according to ACT model standards.²⁵ All staff provided all services, except vocational services, which became the responsibility of the IPS subteam, and specific services limited by licensure to specific professionals. They shared responsibility for every participant’s welfare, and sought to form strong working alliances between each participant and the entire ACT-IPS program. To minimize harm from acute illness episodes, the subteam maintained a very low staff/participant ratio (1:10), and were available on-call 24 hours every day. No pre-determined time limits were placed on service eligibility, regardless of relapse or treatment plan non-adherence.

The IPS subteam adhered to IPS model standards.²⁰ Employment specialists assessed each participant’s past work experiences, current skills, and tolerance for type and intensity of job demands. Participants chose which jobs to pursue, although specialists strongly urged competitive jobs over work adjustment experiences in protected settings. Together participants and specialists searched for competitive job openings, and/or agreed to placement into new jobs “developed” jointly by specialists and local employers. Specialists provided time-unlimited support before, during, and after periods of employment. Like ACT staff, they shared responsibility for meeting every participant’s employment goals.

Comparison traditional program

SWCMHC and the Genesis Center (GC), a local independent non-profit vocational rehabilitation agency specializing in employment of adults with SMI, signed a formal agreement creating the comparison program, which they named the Supported Employment Program (SEP), despite its traditional vocational rehabilitation philosophy. SWCMHC provided mental health and brokered case management services in parallel to GC’s vocational services. To reduce differences in program service intensity between SEP and the larger ACT-IPS program, which might confound service intensity with work outcomes, SWCMHC dedicated two BA-level case managers to serve only SEP participants, at a staff/participant ratio not to exceed 1:30.

The GC program consisted of 10 full-time experienced employment specialists, ranging in education from doctoral degrees to high-school diplomas. About equal numbers were African-American or White. Subteams of specialists, each with a leader and shared caseload, met two to four times per week to review participants’ progress and once per week with the Executive Director for supervision. All specialists developed participant work plans, conducted office-based work-readiness assessments, and provided on-the-job support at a staff/participant ratio at or below 1:10. GC operated 20 hours per day on weekdays, and 4 ½ hours on Saturday.

Although both GC and IPS viewed competitive work as an ultimate goal, IPS immediately aimed for competitive jobs, while GC introduced participants first to graduated work adjustment experiences as preparation for handling competitive job demands. After assessing each participant’s job skills and interests, GC specialists placed participants into one of its temporary, staff-supervised “set-aside” jobs, which differ from competitive jobs in several ways. First, GC contracted for these jobs directly with employers, vesting both with authority for terminating a job against a participant’s wishes. In contrast, participants who contracted with employers in the competitive labor market permitted only the employer to terminate a job against a participant’s wishes. Second, GC’s agreements with employers stipulated that GC, not a participant, held ultimate responsibility for meeting employers’ job performance standards. If a participant failed to meet standards, then either GC staff assigned another participant to the job, or they worked the job until another participant became available to take the job. Third, GC set a 6-month time limit on holding set-aside jobs, after which the jobs turned over to new participants. For participants judged as ready for competitive work, specialists assisted with a brief job search and two weeks of job support. Any participant who failed to hold a competitive job could re-enroll for additional GC work adjustment services per protocol as above.

Fidelity of Implementation

ACT subteam

On a quarterly basis, the ACT consultants rated the ACT subteam’s service performance using the 72-item ACT Fidelity Checklist (yes-no format) covering model standards organized into 13 domains (e.g., outreach and continuity of care, staff functioning as generalists, majority of services delivered in home and community settings).²⁵ Checklist psychometric properties have not yet been reported. Consultants reviewed service and job activity logs, treatment plans, and progress notes; interviewed staff and participants in office and field settings; and observed field interventions. Yearly-averaged checklist scores rose steadily over the project periods: 77%, 80%, 91%, and 94% (% = number of yes answers to the 72 items), for Years 01–04, respectively, indicating high fidelity in Years 03 and 04. In reports and exit meetings with staff, consultants offered specific recommendations to improve service quality.

IPS subteam

On a quarterly basis, the external IPS consultants rated the IPS subteam’s service performance using the 15-item IPS Fidelity Scale. Each item is rated on a 5-point Likert scale ranging from 1 (low) to 5 (high) adherence to each IPS model standard, with behavioral descriptions anchoring each point. Items are allocated to three rationally-defined subscales (a) Staffing (e.g., specialists provide only vocational services), (b) Organization (e.g., specialists integrate vocational with mental health services), and (c) Services (e.g., rapid and individualized job search and placement). The range of total scale scores is 15 to 75 (alpha = .90); total scores clearly discriminate IPS and other SE programs from traditional vocational rehabilitation. To rate fidelity, IPS consultants drew upon identical information sources as did the ACT consultants. Yearly-averaged total scores rose steadily over the project period: 64, 69, 72, and 72, for Years 01–04, respectively, indicating very high IPS model fidelity in Years 02–04. In reports and exit meetings with staff, consultants proposed specific program changes to enhance service quality.

SEP

At the end of Years 02 and 04, the first two authors assessed how closely SEP program services matched those specified in the original study plan. They examined SWCMHC and GC annual service and financial reports, and interviewed staff and both Executive Directors. Staff provided services per study protocol with no significant deviations over the project period.

Measures

Employment outcomes

Both IPS and GC employment specialists recorded attributes of all participants’ jobs, including: (a) duties; (b) who contracted for the job (participant or GC); (c) job status as set aside for persons with SMI or in the open labor market; (d) co-worker disability profiles, ranging from all with disabilities to none; (e) methods used to obtain jobs (e.g., self, job development); (f) available benefits (e.g., insurance); and (g) reasonable accommodations for impairments. Specialists submitted weekly updates for each working participant, including hours worked, income earned, and reasons for job terminations. We classified a job as competitive if the employer (a) paid competitive wages above the federal minimum to participants and workers without disabilities alike; (b) did not set aside the job for adults with disabilities; and (c) located the job in a typical community setting; and if (d) the participant contracted for the job.

Background characteristics and clinical and psychosocial outcomes

RAs conducted face-to-face interviews using the EIDP multisite protocol and assessed demographic characteristics; past psychiatric and substance misuse history; job experience; income sources; federal benefits; and other contextual information at baseline and at months 6, 12, 18, and 24. Participants received $25 for each interview completed. RAs evaluated psychiatric symptoms with the 30-item Positive and Negative Syndrome Scale (PANSS).²⁸ Each item is rated on a 7-point scale from 1 (not applicable) to 7 (extreme). We used White et al’s “pentagonal model”²⁹ containing 25 of the original 30 items, organized into five factor analytically-derived scales. Internal consistency reliability estimates (alpha) for each scale averaged across the five interviews were: Positive (.71), Negative (.79), Autistic Preoccupation (.60), Activation (.70), and Dysphoria (.69). Floor effects on all scales account for relatively low reliability estimates. RAs administered the brief version of the Quality of Life Interview (QOLI)³⁰ concurrently with the PANSS. Participants rated their subjective satisfaction with housing, family relations, social relations, finances, and physical health, on a 7-point Likert scales ranging from 1 (terrible) to 7 (delighted). Coefficient alphas for the five scales averaged across the five interviews ranged narrowly between .75 and .85.

Service use

The multisite EIDP data collection protocol defined 10 vocational and 7 mental health services (exact operational definitions can be retrieved from www.psych.uic.edu/EIDP/Service_Categories.pdf). ACT-IPS and SEP providers submitted daily service logs, noting, for each contact, the service type, need, duration, location, and all persons involved. For program comparisons, we grouped services into vocational and mental health service clusters.

Data Sources, Collection, and Quality Control

In this unblinded community-based implementation effectiveness study, we attempted to minimize data collection bias and contamination, by separating tasks among personnel and limiting their contact with each other. The investigator team trained each RA to administer the EIDP multisite interview protocol, supplemented by ongoing recalibration training provided by the EIDP Coordinating Center and the first author. On a continuing basis, our data entry specialist and data manager reviewed the accuracy of all providers’ service delivery logs and all weekly logs of participant job activity compiled by employment specialists. Blind to study participant, the first author reviewed job logs for logical consistency. The data manager ran biweekly statistical logic and outlier checks to identify remaining problems. He contacted providers to verify data and correct errors, and generated an audit trail of all data base changes. Independent ACT and IPS consultants conducted fidelity assessments, and with the second author, they supervised and mentored ACT-IPS program providers. SEP supervisors from the SWCMHC and GC agencies monitored and mentored its staff.

Power Analysis

From data reported by the two SE randomized clinical trials completed by the time of study start-up, ^{31, 32} we assumed that the average monthly competitive job income earned by their SE ($158 ($245)) and comparison ($54 ($144)) program participants would be reasonable targets for the ACT-IPS and SEP programs, respectively. Using t-tests with alpha set to .05 and power to .80, 59 participants per program were required to rule out sampling error as a plausible explanation for program differences in earned income.

Statistical Analysis Plan

Participant characteristics, services received, and endpoint work outcomes

We compared baseline characteristics of ACT-IPS and SEP participants, using chi-squared difference tests for nominal-scaled variables, and Wilcoxon rank sum tests for ordinal-scaled variables. We profiled service receipt by cumulating total contacts across participants by program over the study period for each individual service and for the vocational and mental health service clusters. For work outcomes, we cumulated values across participants by program over the study period. Highly positively skewed distributions for both services and work outcomes required computation of medians (interquartile ranges); and comparison of program work outcomes with Wilcoxon rank sum tests and effect sizes computed from the U-statistic.³³

Income earned over time

We divided the study period into six 4-month periods to generate enough measurement time-points for testing explanatory effects of time, program, and their interaction on income earned from competitive jobs. Across intervals, income appeared as highly positively-skewed semi-continuous distributions with modes of zero. Percentages of zero values, positive values, and missing data points for the ACT-IPS program were 55.0%, 33.6%, and 11.4%, and for the SEP program were 77.2%, 11.5%, and 11.3%.

Commonly-used linear transformations fail to normalize these distributions to meet assumptions of parametric statistical models. For example, log transformation of earned income normalizes skewed nonzero values, but leaves zero values unaffected. Conducting nonparametric analyses of these distributions by recoding values into categories offers no advantage either. For example, recoding income values into “yes-no” dichotomies for logistic regression analysis prevents addressing important questions about “how much income was earned at what rate over time.” Recoding income into ranks for Wilcoxon tests produces asymmetrical distributions dominated by “ties” at zero values. Thus, we treated income as a mix of two separate distributions of zero and positive values, representing two distinct but correlated “random processes.” These processes describe, “whether or not a participant worked during each of the study’s six intervals,” and conditional on working, “how much income a participant earned at what rate” over these intervals, respectively. The correlation between work probabilities and income amounts may yield information about the nature and pace of participants’ recovery of work capacities; and differential program impacts on work outcomes over time.

We evaluated these two random processes using mixed-effects mixed-distribution models with correlated random effects, which contain two components, one for each random process.³⁴ First, the “logistic” response component estimates the conditional probability of a zero value (working during a given interval or not) as a function of three explanatory fixed effects (time, program, and their interaction) and one between-participants random effect (intercept). Second, conditional on working, the “lognormal” response component estimates the mean of log-transformed positive values of income as a function of the same set of fixed effects specified for the logistic component, plus a between-participants random intercept and one within-participants’ random effect (residuals). We assumed random effects as bivariate normally distributed. A final random effect accounts for the correlation (i.e. link) between the two components’ random intercepts, which yields the strength of association between work probability and income amount per interval.

We conducted these analyses using a SAS Macro, ‘MIXCORR,”³⁵ which sequentially calls in SAS GENMOD and SAS NLMIXED procedures to estimate, by interval, work probability (logistic component), and conditional on working, log-transformed average income (lognormal component), respectively. We centered time on the final interval to assess the full 24-month program main effect on income. Using a model building strategy, we evaluated nested models of progressing complexity. Model 1, “unconditional means (UCM),” contains only one fixed effect (intercept) and serves as the baseline upon which to compare how much the three explanatory fixed effects account for work probabilities and income amounts. Adding a fixed linear time effect to both components generates Model 2, “unconditional growth (UGW).” Introducing fixed effects for study program (ACT-IPS and SEP), and their interaction with time, to the UGW model yields Model 3, “full program-time model (PTM).”

In exploratory analyses (Models 4 – 5), we re-estimated the PTM explanatory effects on income, controlling for time-varying effects of clustered vocational and mental health service contacts by interval in separate analyses. These analyses generated preliminary data upon which we could judge plausibility of our speculations about how the programs facilitated or inhibited participants’ vocational recovery and job success. Although we do know that the two programs functioned very differently in day-to-day practice, we concede that nonspecific effects of simple receipt of any type or amount of service might more parsimoniously account for effects of program (and its interaction with time) on earned income.

In a final set of exploratory analyses (Models 6 – 11), we re-estimated the PTM explanatory effects on income, specifying six time-invariant demographic and diagnostic characteristics as main effects and interactions with program (i.e. potential moderators of program impact) in separate analyses. We selected these characteristics (job history, benefit status, diagnosis, education, race, age, and gender) in advance owing to their common association with work outcomes reported in prior studies.²³

Secondary psychiatric symptom and quality-of-life analyses

Both PANSS factor-derived scale scores and self-reported quality-of-life (QOLI) scale scores were normally distributed. We conducted linear mixed-effect regression models (SAS PROC MIXED) for each scale score specifying the same set of explanatory fixed effects as for the MIXCORR models, and also specified random effects for final status (intercept) and time. Because five interviews were conducted at baseline and at 6-month intervals, time was defined as four 6-month intervals and centered on the final interval, and baseline scores on each scale were specified as covariates.

Results

Characteristics of Participants at Baseline and Attrition Patterns

At baseline, characteristics of both programs’ participants were similar (Table 1). At least 80% of both programs’ participants completed the comprehensive assessment interview at months 6, 12, 18, and 24, with one exception (74% of ACT-IPS participants at month 12). Figure 1 summarizes participants’ study-related activity from random assignment through the 24-month endpoint. Enrollment began in June 1996 and ended in May 1998; follow-up concluded in May 2000. Immediately after program assignment, three SEP and five ACT-IPS participants withdrew. A similar and high percentage of ACT-IPS and SEP participants received vocational services through the first 6 months (86% vs. 83%). Over time, however, a much higher percentage of ACT-IPS participants received vocational services (82% vs. 56% at 12 months; and 67% vs. 26% at endpoint). A similar and high percentage of ACT-IPS and SEP participants received mental health services through 18 months (82% vs. 78%), with modest divergence by study endpoint (79% vs. 69%). Overall, both programs demonstrated steady rates of attrition across the study period, defined as the final month in which a participant received any service and/or worked as reported directly by study providers. The ACT-IPS and SEP programs lost about 7% and 2–3% of their participants per 6-month interval, respectively.. Most ACT-IPS and SEP participants remained active through 18 months (83% vs. 79%), but a modestly higher percentage of ACT-IPS participants completed the study (82% vs. 70%). Voluntary withdrawal accounted for most attrition, followed by relocation outside the service area.

Table 1.

Baseline Characteristics of Study Participants by Program^*

Characteristics	SEP (n = 77)	ACT-IPS (n = 66)	Statistic †	P Value
Age, y			U = 2811 z = 1.03	.29
18–25	9 (11.7)	9 (13.6)
26–45	59 (76.6)	42 (63.6)
> 46	9 (11.7)	15 (22.7)
Female	42 (54.6)	47 (71.2)	χ2 (1) = 4.20	.06
Race			χ2 (2) = 0.10	.95
African American not Hispanic	60 (77.9)	50 (75.8)
White not Hispanic	14 (18.2)	13 (19.7)
Other	3 (3.9)	3 (4.6)
Not married/not cohabitating	63 (81.8)	55 (83.3)	χ2 (1) = 0.06	.83
Educational attainment			U = 2707 z = 0.44	.67
Some high school	38 (49.4)	31 (47.0)
High school diploma/GED	19 (24.7)	15 (22.7)
Some college/technical school	20 (26.0)	20 (30.3)
Monthly income ($)			U = 2275 z = − 1.51	.15
0 – 500	37 (48.1)	42 (63.6)
500 – 750	32 (41.6)	16 (24.2)
> 750	8 (10.4)	8 (12.1)
Mental illness			χ2 (1) = 2.34	.15
Schizophrenia-spectrum	57 (74.0)	41 (62.1)
Mood-spectrum	20 (26.0)	25 (37.9)
PANSS, psychiatric symptoms (M, SD)
Positive (4 items)	7.9 (3.2)	8.3 (3.4)	t (141) = −0.72	.47
Negative (7 items)	17.0 (6.0)	16.9 (5.1)	t (141) = 0.08	.93
Autistic preoccupation (5 items)	11.2 (3.8)	11.4 (4.0)	t (141) = −0.24	.81
Activation (4 items)	7.1 (2.6)	7.6 (3.3)	t (141) = −1.10	.27
Dysphoria (5 items)	12.5 (4.6)	13.1 (5.2)	t (141) = −0.67	.51
Alcohol abuse/dependence (current)	6 (7.8)	7 (10.6)	χ2 (1) = 0.34	.58
Drug abuse/dependence (current)	6 (7.8)	5 (7.6)	χ2 (1) = 0.00	.99
QOLI, self-reported satisfaction (M, SD)
Finances (3 items)	9.5 (4.8)	9.1 (5.2)	t (141) = 0.48	.63
Health (3 items)	13.4 (4.8)	13.8 (4.7)	t (141) = −0.51	.61
Housing (3 items)	13.4 (4.6)	13.5 (5.2)	t (141) = −0.11	.91
Social life (3 items)	15.4 (3.4)	15.8 (3.7)	t (141) = −0.69	.49
Family (2 items)	9.4 (3.2)	9.0 (3.8)	t (141) = 0.70	.48
Paid work past 5 y ‡			U = 2188 z = 0.25	.77
> 0 – 6 mo	30 (40.0)	21 (33.9)
6 – 12 mo	21 (28.0)	23 (37.1)
> 12 mo	24 (32.0)	18 (29.0)
Duration of most recent job §			U = 1490 z = − 1.15	.27
< 1 – 4 mo	28 (39.4)	28 (47.5)
4 – 12 mo	21 (29.6)	18 (30.5)
> 12 mo	22 (31.0)	13 (22.0)
Consecutive mo. unemployed prior to study entry ||			U = 1097 z = 0.54	.61
0 mo	6 (9.7)	3 (5.6)
1 – 12 mo	17 (27.4)	15 (27.8)
> 12 mo	39 (62.9)	36 (66.7)
SSI beneficiary	34 (44.2)	23 (34.8)	χ2 (1) = 1.28	.30
SSDI beneficiary	26 (33.8)	17 (25.8)	χ2 (1) = 1.08	.36
SSDI or SSI beneficiary	50 (64.9)	37 (56.1)	χ2 (1) = 1.17	.31

Abbreviations: SEP, Supported Employment Program; ACT-IPS, Assertive Community Treatment—Individual Placement and Support Program; PANSS, Positive and Negative Syndrome Scale; QOLI, Quality of Life Interview; SSI, Supplemental Security Income; SSDI, Social Security Disability Income

^*Data are given as number (percentage) of participants, except where indicated. Numbers vary due to missing data.

^†Fisher’s Exact Test used for Chi-Squared Tests; Exact Test used for Wilcoxon Rank Sum Tests.

^‡Data missing for 2 SEP and 4 ACT-IPS participants.

^§Conditional on past history of working; data missing for 3 SEP and 2 ACT-IPS participants.

^||Data missing for 15 SEP and 12 ACT-IPS participants.

Figure 1.

Participant Flow—Random Assignment through End of 24-Month Participation Period

Program Service Profiles and Service Integration

Vocational and mental health services

The programs’ vocational service profiles differed considerably in emphasis and diversity (Table 2). More than two-thirds of ACT-IPS participants received 7 of the 10 EIDP-defined vocational services; over half received 9 of 10. However, only a simple majority of SEP participants received 2 of 10. Cumulated over the study period, ACT-IPS participants logged many more vocational service contacts (Mdns = 54 vs. 9) distributed across far more study months (57% vs. 22%). The programs’ mental health service profiles differed less compared to vocational services. Over two-thirds of ACT-IPS participants received 4 of the 8 EIDP-defined mental health services; about half received 6 of 8 (Table 2). More than two-thirds of SEP participants received 3 of 8, but few received other mental health services. Cumulated over the study period, ACT-IPS participants logged more mental health service contacts (Mdns = 86 vs. 59) distributed across slightly more study months (78% vs. 67%) and their grand total of service contacts far exceeded those of SEP participants (Mdns = 140 vs. 83) distributed across slightly more study months (83% vs. 73%). No “contamination” of program service receipt was evident: no SEP participants received services from the ACT-IPS program; no ACT-IPS participants received services from SEP case managers, but on rare occasions, they received SWCMHC mental health services when ACT-IPS providers were unavailable. Only 5 ACT-IPS participants requested a placement into a GC set-aside jobs.

Table 2.

Service Profile for each Program ^*

	SEP			ACT-IPS
Service	n (%)	Mdn	IQR	n (%)	Mdn	IQR
	Vocational services
Vocational assessment	13 (17)	0	0 – 0	59 (89)	3	1 – 4
Job development or finding	11 (14)	0	0 – 0	60 (91)	5.5	2 – 12
Collaboration-employers	28 (36)	0	0 – 1	46 (70)	3	0 – 5
Vocational support groups	0 (0)	0	0 – 0	14 (21)	0	0 – 0
Collaborate-family/friends	13 (17)	0	0 – 0	34 (52)	1	0 – 2
Vocational treatment planning	63 (82)	2	1 – 4	57 (86)	5	2 – 11
Skills training	37 (48)	0	0 – 3	50 (76)	3	1 – 7
Vocational counseling	12 (16)	0	0 – 0	56 (85)	6	2 – 14
Job Support-on site	26 (34)	0	0 – 14	35 (53)	1	0 – 2
Transportation	20 (26)	0	0 – 1	56 (85)	10	3 – 21
All vocational service contacts	69 (90)	9	2 – 40	61 (92)	54	26 – 71
	Mental health services
Case management	74 (96)	26	16 – 43	61 (92)	44	29 – 84
Family/couples counseling	0 (0)	0	0 – 0	6 (9)	0	0 – 0
Emergency care	19 (25)	0	0 – 0	40 (61)	1	0 – 4
Psychiatric evaluation	60 (78)	2	1 – 5	60 (91)	8	4 – 11
Individual counseling	33 (43)	0	0 – 2	46 (70)	2	0 – 5
Group counseling	13 (17)	0	0 – 0	33 (50)	1	0 – 2
Medication management	69 (90)	15	6 – 28	60 (91)	22	11 – 36
Partial hospitalization	2 (3)	0	0 – 0	1 (2)	0	0 – 0
All mental health service contacts	74 (96)	59	31 – 87	61 (92)	86	64 – 157
All service contacts	74 (96)	83	40 – 134	61 (92)	140	96 – 230

Abbreviations: SEP, Supported Employment Program; ACT-IPS, Assertive Community Treatment—Individual Placement and Support Program; IQR, Interquartile range.

^*The sample size is 77 for SEP and 66 for ACT-IPS. Number (percentage) indicates participants who received services

Integration of vocational with mental health services

IPS employment specialists coordinated vocational assessment, planning, counseling, and job development/finding with ACT interventions reducing psychiatric symptoms impairing participants’ job performance. To maximize time and energy that participants could devote to work, ACT staff quickly addressed pressing housing, medical, and other basic needs. When necessary, IPS specialists arranged travel to work sites. With participants’ permission, specialists enlisted their natural support networks (e.g., families and friends) and employers to aid vocational development. Over time, the ACT-IPS program adjusted services to match participants’ evolving job aspirations, enhance acquired job skills, and build enthusiasm to pursue new jobs after loss of a job..

Operating from a more narrow vocational development approach, GC employment specialists prepared participants for competitive work, by placing them in closely-supervised set-aside jobs and concentrating on vocational planning and job skill development. In parallel, SEP case managers brokered mental health and support services, while SEP clinicians evaluated psychiatric status and managed psychotropic medications. Numerous obstacles prevented SEP providers from coordinating vocational with mental health services: they worked out of various locations on different schedules, reported to different supervisors, dealt with differing reimbursement policies, and provided services based on different therapeutic orientations.

Job Outcomes: Endpoint Analyses

Employment rates and income earned

We report outcomes for both competitive and all jobs for two reasons. First, most constituencies agree that, for persons with SMI and limited work histories, holding competitive jobs may increase chances for recovering valued adult roles, and greater community involvement.^{36, 37} Second, advancing toward economic self-sufficiency depends upon earning income, regardless of job type. Of the 48 ACT-IPS working participants, 42 (87%) obtained competitive jobs. In contrast, of the 51 SEP working participants, only 20 (39%) held competitive jobs; the first job landed by 9 of this 20 was a competitive job they obtained on their own. Only 11 of 20 transitioned from an SEP set-aside job into a competitive job. Overall, over twice the percentage of ACT-IPS participants obtained a competitive job (63.6% vs. 26.0%, P < .001, ES = 0.38), but differences for all jobs were less (72.7% vs. 66.2%, P = .40, ES = 0.07; Table 3). ACT-IPS participants earned much more income from competitive jobs, (Mdn = $549, interquartile range (IQR) = $0 – $5145 vs. Mdn = $0, IQR = $0 – $40, P < .001, ES = 0.70), but differences for all jobs were less. (Mdn = $1337, IQR = $0 – $5495 vs. Mdn = $743, IQR = $0 – $2800, P = .07, ES = 0.59).

Table 3.

Cumulated Employment Outcomes by Program.

	Competitive Jobs									All Jobs
Outcome	SEP			ACT-IPS						SEP			ACT-IPS
	n	N	%	n	N	%	χ2(1)	P Value	ES	n	N	%	n	N	%	χ2(1)	P Value	ES
No. who worked	20	77	26.0	42	66	63.6	20.5	<.001	0.38	51	77	66.2	48	66	72.7	0.71	.40	0.07
Jobs by program	35	124	28.0	89	124	71.2	---	---	---	92	203	45.3	111	203	54.7	---	---	---
	Mdn	IQR		Mdn	IQR		U / z	P Value	ES	Mdn	IQR		Mdn	IQR		U / z	P Value	ES
Jobs per worker	2	(1 – 2)		2	(1 – 3)		---	---	---	1	(1 – 2)		2	(1 – 3)		---	---	---
Total earned income
All †	0	(0–40)		549	(0–5145)		3551/ 4.52	.001	0.70	743	(0–2800)		1337	(0–5495)		2983/ 1.81	.07	0.59
With jobs ‡	1857	(747–4443)		2855	(900–6968)		327/ 0.79	.43	0.39	1954	(743–4291)		3388	(1068-8567)		1501/ 1.93	.06	0.61
Total weeks worked
All	0	(0–1)		8.5	(0–41)		3541/ 4.47	.001	0.70	10	(0–43)		16	(0–51)		2804/ 1.08	.28	0.55
With jobs	22.5	(12–44.5)		31.5	(9–53)		316/ 0.64	.52	0.38	25	(10–56)		33.5	(14.5–65.5)		1322/ 0.68	.50	0.54
Total hours worked
All	0	(0–8)		108	(0–800)		3549/ 4.51	.001	0.70	171	(0–601)		296	(0–1032)		2908/ 1.51	.13	0.57
With jobs	360	(146–844)		485	(139–1139)		325/ 0.77	.44	0.39	434	(171–914)		602	(212–1356)		1426/ 1.41	.16	0.58
Weeks/job §	11.5	(7.8–23.5)		12.5	(6.8–22.2)		261/ 0.17	.86	0.31	14	(9–27)		13	(7–27)		1116/ 0.75	.46	0.46
Weeks/longest job §	20	(10–31)		19	(8–39)		420/ 0.00	.99	0.50	21	(10–40)		19	(9–41)		1201/ 0.16	.88	0.49
Weeks to first job §	46	(31–70)		19	(7–39)		616/ 2.95	.005	0.73	4.4	(0–32.7)		19	(7– 39)		1638/ 2.90	.005	0.67
Hours/week/job §	20	(9–27)		22	(14–33)		343/ 1.15	.25	0.41	16	(11–20)		21	(14–32)		1593/ 2.58	.005	0.65
Wage/job ($/h) §	5.50	(5.13–5.60)		5.33	(5.03–6.00)		410/ 0.14	.89	0.49	4.75	(3.56-5.19)		5.15	(4.83–5.56)		1742/ 3.62	<.001	0.71

Abbreviations: SEP, Supported Employment Program; ACT-IPS, Assertive Community Treatment—Individual Placement and Support Program; ES, effect size; ---, not applicable; IQR, Interquartile range.

^*The sample size is 77 for SEP and 66 for ACT-IPS. ES for chi-squared tests are phi coefficients. ES for Wilcoxon rank sum tests derive from the U-statistic.³³

^†“All” indicates all participants randomly allocated to each of the two programs.

^‡“With jobs” indicates participants that worked during the 24-month study period.

^§Conditional on working

Two related factors account for the decrease in effect sizes for the ITT sample on income earned from competitive jobs (0.70) versus all jobs (0.59). First, 80% (89/111) of jobs held by the 48 working ACT-IPS participants were competitive, twice that of 38% (35/89) held by the 51 working SEP participants (Table 3). Thus, ACT-IPS participants earned a higher proportion of their total income from competitive jobs, whereas SEP participants earned a higher proportion from non-competitive jobs. Second, variation of earned income (interquartile ranges) between ACT-IPS and SEP participants were much higher for competitive jobs (IQRs = $0 – $5145 vs. $0 – $40) compared to all jobs (IQRs = $0 – $5495 vs. $0 – $2800), despite that their respective medians increased about equally (Mdns = $549 to $1337 vs. $0 to $743).

Weeks and hours worked

At competitive jobs, ACT-IPS participants worked more weeks and hours, but program differences decreased for all jobs by similar magnitudes as for income (Table 3). Conditional on working at competitive jobs, ACT-IPS participants logged similar weeks per job, weeks per longest job, and hours per week per job at about equal wage rates (Table 3). They obtained their first competitive job much earlier in the study period (Mdns = 19 vs. 46 weeks, P = .005, ES = 0.73), but their first paid job of any kind much later (Mdns = 19 vs. 4.4 weeks, P = .005, ES = 0.67), primarily due to immediate placement of SEP participants’ into GC set-aside jobs. Small to moderate effect sizes (< 0.60) for most program subgroup comparisons failed to reach statistical significance due to lack of statistical power.

Job features

Compared to GC specialists, IPS specialists placed relatively more its participants into competitive jobs developed jointly with local employers (16% vs. 3%), and helped participants find relatively more existing job openings (29% vs. 6%). Relatively fewer ACT-IPS compared to SEP participants obtained competitive jobs on their own (38% vs. 60%) and through informal contacts (17% vs. 31%). Overall, the ACT-IPS program aided a much higher proportion of its participants to get competitive jobs (45% vs. 9%), reflecting the IPS assertive, long-term approach to competitive work.

Very few competitive jobs came with benefits: less than one in five jobs offered any insurance; only one in four granted any paid leave. Most job functions, defined by the “Dictionary of Occupational Titles (DOT),” ³⁸ required few technical skills. Competitive jobs were equally distributed across three of nine DOT occupational groups: “service” (e.g., house keeping), “processing” (e.g., poultry products), and “structural work” (e.g., home construction).

Job terminations

Almost half of both programs’ participants quit competitive jobs before securing a new job; employers laid off or fired about a third. IPS and GC employment specialists determined reasons for job losses from discussions with participants, and when they granted permission, employers and natural support networks. Reasons for job losses are reported for the job as the unit of analysis; multiple reasons were cited for about 25% of jobs. Because one SEP participant and 12 ACT-IPS participants continued working in competitive jobs at study endpoint, we report on 34 of 35 SEP jobs and 77 of 85 ACT-IPS jobs. Loss of ACT-IPS and SEP competitive jobs, respectively, due to quitting or firing, were primarily associated with symptoms impairing job performance (11.7% and 8.8%); job pressures (11.7% and 2.9%); inability to perform job tasks (13.0% and 5.9%); dissatisfaction with job duties (9.1% and 5.9%); and discontinuation of job positions (9.1% and 23.5%). Very few quit jobs out of dissatisfaction with schedules, hours, pay rate, or discrimination. Few were fired due to conflicts with coworkers and/or supervisors. Surprisingly, only 1 of 31 workers receiving federal benefits quit competitive jobs from fear that earning income would lead to benefits termination.

Job Outcomes: Time Trends of Income Earned from Competitive Jobs

Descriptive statistics

Percentages of ACT-IPS participants working in competitive jobs by 4-month intervals ranged narrowly between 33% and 38% throughout the study period. In contrast, much lower percentages of SEP participants worked by interval, ranging from less than 10% during the first three intervals, reflecting their early involvement in GC set-aside jobs, to a high near 20% in the final interval,. Conditional on working, ACT-IPS participants earned more income per interval throughout the study period compared to SEP participants, although differences varied inconsistently over time between small in the second interval (Mdn = $1099, IQR = $670 – $2635 vs. Mdn = $930, IQR = $190 – $1140) and moderately large in the fifth interval (Mdn = $2084, IQR = $1156 – $3455 vs. Mdn = $1284, IQR = $300 – $1972).

Primary regression models

We evaluated the effects of the three explanatory variables, program, time, their interaction, on the probability of working and the amount of income earned per interval, with a sequence of nested mixed-effects, mixed-distribution regression models. Model 1, the unconditional means model (UCM), yields, for all participants regardless of program and time, the predicted average probability of working a competitive job (logistic intercept = −2.35, P < .001) and the average amount of income earned (lognormal intercept = 5.58, P < .001; Table 4). The UCM’s parameter estimates and fit indices provide a baseline to judge impact of the three explanatory variables on income earned over time. Model 2, the unconditional growth model (UGW), shows, for all study participants, a considerable increase in the probability of working over time (logistic time = 0.32, P < .001), but only a slight increase in earned income over time (lognormal time = 0.14, P < .001; Table 4). The explanatory effect of time markedly improves model fit.

Table 4.

Work Probabilities and Income Earned from Competitive Jobs over Time by Program

	Model 1 Unconditional means		Model 2 Unconditional growth		Model 3 Program x Time		Model 4 Vocational services †		Model 5 Mental health services †
	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value
	Probability of Working (Logistic Component)
Intercept	− 2.35 (0.33)	<.001	− 1.65 (0.37)	<.001	− 2.13 (0.52)	<.001	− 2.31 (0.55)	<.001	− 2.34 (0.56)	<.001
Time			0.32 (0.07)	<.001	0.74 (0.14)	<.001	0.80 (0.15)	<.001	0.77 (0.14)	<.001
Program (ACT-IPS=1, SEP=0)					1.31 (0.65)	.046	1.29 (0.67)	.057	1.28 (0.67)	.06
Program x Time interaction					− 0.63 (0.17)	<.001	− 0.63 (0.17)	<.001	− 0.62 (0.17)	<.001
Covariate ‡							0.031 (0.013)	.015	0.016 (0.010)	.11
Random intercept	6.71 (1.74)	<.001	7.52 (1.95)	<.001	6.35 (1.67)	<.001	6.88 (1.88)	<.001	6.82 (1.84)	<.001
	Income from Competitive Job (Lognormal Component)
Intercept	5.58 (0.25)	<.001	5.86 (0.26)	<.001	5.30 (0.39)	<.001	5.32 (0.39)	<.001	5.23 (0.40)	<.001
Time			0.14 (0.04)	.001	0.15 (0.10)	.11	0.14 (0.10)	.15	0.16 (0.10)	.10
Program (ACT-IPS=1, SEP=0)					1.10 (0.38)	.005	1.12 (0.38)	.003	1.11 (0.39)	.005
Program x Time interaction					− 0.009 (0.11)	.93	− 0.013 (0.11)	.90	− 0.002 (0.11)	.99
Covariate ‡							− 0.006 (0.01)	.39	0.004 (0.005)	.51
Residual (within-participants)	0.79 (0.10)	<.001	0.72 (0.09)	<.001	0.72 (0.09)	<.001	0.72 (0.09)	<.001	0.71 (0.09)	<.001
Random intercept	1.47 (0.48)	.002	1.56 (0.49)	.002	1.38 (0.43)	.002	1.33 (0.43)	.002	1.44 (0.46)	.002
-2LL / AIC	1200	1215	1168	1184	1128	1152	1121	1149	1126	1154
Selected Models (7 – 12)	Job months-past 5 y §		SSI/SSDI ||		Diagnosis ¶		Education level §		Race #
Parameter	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value
	Probability of Working (Logistic Component)
Intercept	− 2.41 (0.55)	<.001	− 0.70 (0.79)	.38	− 2.77 (0.62)	<.001	− 1.98 (0.53)	<.001	− 2.52 (0.58)	<.001
Time	0.72 (0.14)	<.001	0.74 (0.14)	<.001	0.74 (0.14)	<.001	0.74 (0.14)	<.001	0.74 (0.14)	<.001
Program (ACT-IPS=1, SEP=0)	1.08 (0.67)	.11	0.76 (1.05)	.47	1.60 (0.77)	.039	1.23 (0.66)	.06	1.48 (0.71)	.040
Program x Time interaction	− 0.61 (0.16)	<.001	− 0.63 (0.17)	<.001	− 0.63 (0.17)	<.001	− 0.64 (0.17)	<.001	− 0.63 (0.17)	<.001
Covariate	0.051 (0.022)	.025	− 1.87 (0.87)	.033	2.25 (0.86)	.010	0.35 (0.33)	.28	2.02 (0.95)	.034
Covariate x Prog. interaction	0.014 (0.029)	.63	0.69 (1.16)	.55	− 1.24 (1.11)	.27	− 0.13 (0.41)	.75	− 0.77 (1.28)	.55
Random intercept	4.88 (1.33)	<.001	5.86 (1.56)	<.001	5.80 (1.54)	<.001	6.23 (1.64)	<.001	5.95 (1.58)	<.001
	Income from Competitive Job (Lognormal Component)
Intercept	5.26 (0.41)	<.001	6.44 (0.44)	<.001	4.69 (0.45)	<.001	5.43 (0.38)	<.001	5.25 (0.43)	<.001
Time	0.16 (0.10)	.10	0.17 (0.09)	.07	0.18 (0.09)	.06	0.18 (0.09)	.06	0.15 (0.10)	.13
Program (1=ACT-IPS, 0=SEP)	0.92 (0.41)	.026	0.49 (0.51)	.34	1.53 (0.46)	.001	1.00 (0.38)	.010	1.03 (0.43)	.017
Program x Time interaction	− 0.018 (0.11)	.86	− 0.03 (0.11)	.78	− 0.03 (0.11)	.74	−0.034 (0.11)	.74	−0.004 (0.11)	.97
Covariate	0.022 (0.013)	.097	− 1.45 (0.45)	.002	1.66 (0.49)	.001	0.46 (0.22)	.036	0.55 (0.53)	.30
Covariate x Prog. interaction	0.006 (0.016)	.71	0.77 (0.56)	.17	− 1.18 (0.60)	.051	− 0.25 (0.25)	.31	0.10 (0.67)	.88
Residual (within-participants)	0.72 (0.09)	<.001	0.74 (0.10)	<.001	0.72 (0.09)	<.001	0.70 (0.09)	<.001	0.72 (0.09)	<.001
Random intercept	1.05 (0.34)	.002	1.02 (0.34)	.004	1.24 (0.39)	.002	1.37 (0.42)	.002	1.25 (0.40)	.002
-2LL / AIC	1111	1143	1115	1147	1114	1146	1121	1153	1121	1153

Abbreviations: SEP, Supported Employment Program; ACT-IPS, Assertive Community Treatment—Individual Placement and Support Program; Est., Estimate; -2LL, minus twice the LogLikelihood. AIC, Akaike Information Criterion.

^*Time is defined as six 4-month intervals and centered on the last interval. The sample size is 77 for SEP and 66 for ACT-IPS. Smaller values for -2LL and AIC indicate better model fit.

^†Number of contacts per interval.

^‡Covariate = vocational and mental health services and demographic and diagnostic characteristics entered into Model 3 (Program x Time).

^§Centered at each program’s median.

^||SSI/SSDI (1 = Yes, 0 = No).

^¶Diagnosis (1 = Mood-spectrum, 0 = Schizophrenia-spectrum).

^#Race (1 = Caucasian, 0 = African American).

Model 3, the “full program-time model (PTM)” reveals that, although ACT-IPS participants were much more likely to work during the final interval (logistic program = 1.31, P < .001), differences in work probabilities between programs decreased markedly from the first to the last interval (logistic program by time interaction = − 0.63, P = .001; Table 4). Conditional on working, ACT-IPS participants earned modestly more income (lognormal program = 1.10, P = .005), but the relative difference between programs did not change over time (lognormal interaction = − 0.01, P = .93). Statistically significant random intercepts for both logistic and lognormal components (6.35 and 1.38, respectively, Ps < .001) indicate that explanatory fixed effects for time, program, and their interaction leave “unexplained” much between-participant variation in work probabilities and income earned over time, despite improved model fit.

The high positive correlation between the PTM’s logistic and lognormal random intercepts (r = 0.92, P < .001), indicates that, as the percentage of ACT-IPS and SEP participants working per interval increased over time, on average, they earned more income per interval over time at about the same rate. Were this correlation negative, then a decreasing percentage of participants would have worked per interval over time, but would have, however, earned more income per interval over time, a good outcome for few participants, but a poor program outcome.

Secondary regression models

For vocational service contacts, both ACT-IPS and SEP participants logged the most in the first interval (Mdn = 13, IQR = 5 – 22 vs. Mdn = 2, IQR = 0 –11) and the least in the final interval (Mdn = 5, IQR = 0 – 9 vs. Mdn = 0, IQR = 0 – 1). For mental health service contacts, both ACT-IPS and SEP participants logged the most in the first interval (Mdn = 44, IQR = 22 – 62 vs. Mdn = 15, IQR = 8 – 22) and the least in the final interval (Mdn = 10, IQR = 2 – 10 vs. Mdn = 4, IQR = 0 – 10). Controlling for time-varying effects of vocational and mental health service contacts did not substantively alter magnitude or direction of the PTM’s explanatory effects on work probabilities and income earned over time (Table 4).

Four of six participant characteristics were substantively associated with work probability; three of six were substantively associated with income earned. However, lack of statistically significant interactions with program rules out these characteristics as moderators of program impact in this study sample and setting (Table 4). Characteristics associated with lower likelihood of working during the study period were race (African American), diagnosis (schizophrenia), work in the past five years (fewer months), and federal benefits (currently on the rolls). Conditional on working, characteristics associated with less income earned during the study period were diagnosis (schizophrenia), educational attainment (less), and federal benefits (currently on the rolls).

For all jobs, a parallel set of identically-specified models yielded similar program patterns of work probabilities and income earned over time, but smaller program differences compared to competitive jobs (results not shown, but available from the authors).

Secondary Symptom and Quality-of-Life Outcomes

We found no substantive differences between both programs’ participants and no substantive change from baseline over time for psychiatric symptoms (PANSS scale scores) and self-reported quality-of-life (QOL-I scale scores) (results not shown but available from the authors). Floor effects (i.e. less than mild symptoms) probably account for null findings across all PANSS scale scores. Low prevalence of substance use disorders (< 10%) at baseline did not change over time. Adverse events included psychiatric hospitalization (SEP, n = 21, 27%; ACT-IPS, n = 17, 26%); a period of homelessness for one SEP and one ACT-IPS participant; and sentencing of three SEP and three ACT-IPS participants to prison. A causal relationship between these events and study participation could not be demonstrated.

Discussion

Summary of Implementation Problems

We addressed five questions in this study. First, in a rural South Carolina county, we encountered formidable difficulties with implementing evidence-based model programs that integrate vocational rehabilitation with specialty mental health services. Recruiting and retaining the necessary numbers of qualified providers to staff the new ACT-IVR and IPS programs proved to be insurmountable. However, merging these two partially implemented programs formed the fully-staffed ACT-IPS program, whose work outcomes for rural adults with SMI and limited employment experience compared favorably to urban-based SE programs. Maintaining ACT-IPS program effectiveness throughout the project period required ongoing training, mentoring, and fidelity monitoring. Upon termination of federal grant support, the South Carolina Department of Mental Health picked up the funding to sustain the ACT-IPS program into the present.

Summary of Competitive Work Outcomes

Second, high participation rates through study endpoint and lack of marked differential participant attrition by program strengthens validity of program comparisons. Across a set of competitive work outcomes, the ACT-IPS integrated service program outperformed the SEP parallel service program. Compared with SEP participants over the 24-month study period, more than twice the percentage of ACT-IPS participants held competitive jobs (64% vs. 26%). Conditional on working in competitive jobs, they earned about 1.5 times the median income ($2855 vs. $1857), and worked just under 1.5 times the median hours and weeks. Integrating and orienting all services to enhance participants’ job performance and marketability are superior to parallel services lacking an employment focus, a straight-forward inference consistent with findings from two recent SE process analyses reporting strong positive associations between service integration and competitive work outcomes.^{39, 40} Moreover, providing integrated services in participants’ home and community settings probably facilitates productive long-term program-participant collaborations, by overcoming logistical and geographic impediments to accessing a full-range of rural-based services.^{12, 17}

Effectiveness of Different Strategies for Obtaining Competitive Work

Third, although the SEP program eliminated barriers to work by placing participants immediately into its time-limited set-aside jobs, and re-enrolled any participant subsequently unable to hold a competitive job, the program did not increase its participants’ competitive work activity over the study period. Of the 51 SEP participants who worked, only 20 held competitive jobs. Only 11 of these 20 first worked in a set-aside job and then in a competitive job. Despite guaranteed access to set-aside jobs, SEP participants still earned and worked less in all jobs taken together than ACT-IPS participants in a rural labor market with few work opportunities. Our rural work outcomes replicate findings of prior urban-based trials showing graduated work adjustment as an ineffective strategy for increasing competitive job tenure for most adults with SMI and little job experience.²²

Patterns of Competitive Work over Time

Fourth, work patterns varied markedly across participants and over time. One-third of participants did not hold a job during the study period. Participants who worked manifested wide variation in job tenure, and time taken to obtain their first competitive and any paid job. Such heterogeneous work patterns resist easy explanation. Each participant’s initiation, pace, and nature of recovering work capacities may follow a unique and nonlinear trajectory over time, independent of service receipt.⁴¹ For those who did receive considerable services over extended periods, service impacts on work activity might lag differentially across participants over time. However, uneven work patterns might also be partly a function of participants’ employment goals, which ranged from little or no work to full-time work expected by federal social assistance programs and recent phase-ins of Ticket-to-Work and Work Incentives Improvement Act (1999, Public Law 106–170) provisions. Because factors associated with dissimilar work trajectories over time are not well known, we recommend identifying potent personal, service, and environmental contextual factors that may drive and/or inhibit, vocational recovery paths over time.⁴² Findings will help employment specialists to better match selection and timing of services to each person’s unique and evolving recovery status, strengths, and work preferences.

Association of Service and Participant Characteristics with Competitive Work Outcomes

Fifth, controlling for the time-varying effects of total vocational and mental health contacts did not substantively alter effects of time, program, and their interaction on competitive work probabilities and income earned over time. However, “total service contacts” is a blunt time-varying measure that does not capture service integrative functions, as described in prior SE process analyses, and may therefore lack construct validity.

Two of six pre-selected participant characteristics, diagnosis (schizophrenia) and federal benefit status (currently on the rolls) were associated with the lowest likelihood of working and the lowest income over time, consistent with findings of prior studies. Because no interactions between the six characteristics and program on work probabilities and earned income achieved statistical significance, they cannot be inferred, in this study sample and setting, as moderators of program impact.

African-American participants, representing 75% of the sample, were half as likely to work compared to Whites after accounting for the explanatory effects. Conditional on working, however, both African Americans and Whites earned about the same amount of income. Nevertheless, African Americans have historically faced the severest educational, economic, and health disadvantages in the study area. Developing innovative strategies to help them overcome hardships not of their own making, and realize employment goals, must become a policy priority.

Comparison of Competitive Work Outcomes with Published SE Trials

Of the 14 completed randomized clinical trials, individual reports of eight have been published. ^{31, 32, 45} All eight trials reported percentages of SE and comparison program participants who held at least one competitive job during 12- to 24-month study periods. SE program percentages ranged from 27% to 78% compared to 64% in our study. Comparison program percentages ranged from 7% to 40% compared to 37% in our study. Five trials^{32, 45} reported income differences between SE and comparison programs as means (SDs), requiring that we shift from medians (IQRs) to this metric. SE program earnings per month ranged from $37 to $189 compared to $114 in our study. Comparison program earnings per month ranged from $3 to $60 compared to $39 in our study. Thus, the magnitude of the ACT-IPS program’s work probabilities and monthly earned income fell between the median and maximum values of the published trials’ SE and comparison programs, despite operating in a rural economy with limited job opportunities, high service sector unemployment, and high poverty among African Americans. In line with the eight trials, we found no substantive differences between our study programs on psychiatric symptoms and self-reported quality of life. Because our symptom measure manifested floor effects, we cannot accurately estimate program impact on symptoms.

Limitations of Inference

First, project redesign and deviation from a pre-specified random assignment process may have compromised study internal validity and program construct validity. Although sensitivity analyses revealed no substantive differences across a wide spectrum of participant and program variables before and after redesign, we cannot rule out that selection biases, program changes, and unobserved variables confounded between-program comparisons. Second, we did not measure some key constructs associated with work outcomes, particularly working alliance ⁴⁹ and neurocognitive status, both of which may account for some “unexplained” participant-level variance on competitive work outcomes. Third, competitive work outcomes of the ACT-IPS program, which integrated vocational and mental health services within a self-contained team, may not be directly comparable to those of free-standing SE programs that integrate their vocational services with independent CMHC mental health services.

Conclusions

A single program that blended two evidence-based practices (ACT and IPS) and tightly integrated vocational rehabilitation with mental health services functioned respectably in a rural service mix and economy. The ACT-IPS program aided persons with SMI-related work impairments, limited job experience, and tremendous educational and economic disadvantage, to attain competitive work outcomes on a par with the urban SE programs previously described in the literature. Moreover, the program’s service coordinating functions and immediate attention to competitive work may have been key processes that overcame common rural area obstacles to receipt of appropriate services and to labor market participation. However, the median participant in this and other SE trials worked 5 to 10 full-time equivalent weeks of work per year, earning between $1000 and $2000 at an hourly wage at or near the federal minimum. To be sure, these outcomes reflect meaningful gains in labor market activity over one- to two-year intervention periods, but they fall far short of economic independence. (2.27) We firmly believe that furthering work activity requires public policy adjustments that reduce barriers to accessing higher education,⁵⁰ career-oriented jobs providing opportunities to master highly marketable “technical skills,”⁵¹ and, for those who leave federal insurance programs to work, rapid restoration of benefits eligibility in the event of setbacks during ongoing recovery efforts.