Randomized Trial of Supported Employment Integrated With Assertive Community Treatment for Rural Adults With Severe Mental Illness

Abstract

Urban-based randomized clinical trials of integrated supported employment (SE) and mental health services in the United States on average have doubled the employment rates of adults with severe mental illness (SMI) compared to traditional vocational rehabilitation. However, studies have not yet explored if the service integrative functions of SE will be effective in coordinating rural-based services that are limited, loosely linked, and geographically dispersed. In addition, SE's ability to replicate the work outcomes of urban programs in rural economies with scarce and less diverse job opportunities remains unknown. In a rural South Carolina county, we designed and implemented a program blending Assertive Community Treatment (ACT) with an SE model, Individual Placement and Support (IPS). The ACT-IPS program operated with ACT and IPS subteams that tightly integrated vocational with mental health services within each self-contained team. In a 24-month randomized clinical trial, we compared ACT-IPS to a traditional program providing parallel vocational and mental health services on competitive work outcomes for adults with SMI (N = 143; 69% schizophrenia, 77% African American). More ACT-IPS participants held competitive jobs (64 versus 26%; p < .001, effect size [ES] = 0.38) and earned more income (median [Mdn] = $549, interquartile range [IQR] = $0–$5,145, versus Mdn = $0, IQR = $0–$40; p < .001, ES = 0.70) than comparison participants. The competitive work outcomes of this rural ACT-IPS program closely resemble those of urban SE programs. However, achieving economic self-sufficiently and developing careers probably require increasing access to higher education and jobs imparting marketable technical skills.

Introduction

In the United States, the point prevalence of competitive employment for adults with severe mental illness (SMI), work impairments, and limited job experience ranges between 10 and 20%.^1–2 Fourteen U.S.-based randomized clinical trials of supported employment (SE) have on average doubled these rates. Meta-analysis of the first 6 trials reports relative equivalence across SE programs on multiple competitive work outcomes (e.g., employment rates, weeks worked, and income earned) and a consistent two- to threefold improvement over traditional vocational rehabilitation.³ The second 8 trials, constituting the federally sponsored “Employment Intervention Demonstration Program,” replicated these work outcomes.⁴ Based on the cumulative findings of SE program effectiveness, federal agencies and private foundations have recently funded SE adoption efforts throughout U.S. public mental health systems.⁵

Most SE trials have taken place in large U.S. urban areas. Verifying SE services as effective in diverse rural settings requires additional studies addressing at least 2 crucial questions. First, can urban SE program strategies that coordinate vocational with specialty mental health services^6–7 be effectively implemented into underresourced and fragmented rural service arrays? Second, and more important, will rural-based SE programs achieve similar work outcomes as their urban counterparts in southeastern U.S. rural economies, characterized by limited job opportunities, high unemployment, and concentrations of severely disadvantaged African American residents?

Service Coordination

Although rural service profiles vary enormously across U.S. regions,⁸ implementing SE programs' coordinating functions into loosely arrayed, weakly linked, understaffed, and geographically dispersed rural services mixes is extremely difficult.^9–10 In rural South Carolina, the site of our study, obstacles to integrating SE with mental health services may be more challenging to overcome compared to the case in other U.S. rural regions. First, South Carolina's government operates public services with a very small budget relative to most other states. Second, because a much higher proportion of the state's population resides in rural areas (40%) compared to the United States as a whole (21%), the state faces overwhelming challenges to staff rural-based health and social services for persons scattered over large geographic expanses.

Third, African Americans make up a very high proportion of the state and study area's population (30 and 47%, respectively). They experience a disproportionately high prevalence of multigenerational poverty and chronic medical illness compared to other racial/ethnic groups.^11–12 Even worse, African American adults with SMI use mental health services much less than other racial/ethnic groups.¹³ Their low service use and large rural presence have severely exceeded South Carolina's capacity to develop accessible vocational, mental health, and other services responsive to their needs and preferences. Insufficient psychiatric treatment combined with harsh social and economic disadvantage, here and in other southeastern rural regions, probably accounts for African Americans' poorer psychiatric status and functioning compared to other racial/ethnic groups.^{12, 14} Thus, the coordination of rural SE and mental health services not only may considerably improve their accessibility but may also induce synergistic therapeutic effects that alleviate illness and promote labor market success.

Employment Outlook

Job opportunities in southeastern rural areas are limited for adults in general¹² and especially for those with SMI. First, poorly resourced secondary schools grapple with graduating their students. More than one-third of rural African American and one-quarter of rural white students do not complete high school.¹⁵ During their high school years, students who experience onset of prodromal and/or full psychiatric syndromes or who exhibit risk factors for future illness may be less likely to graduate. However, few studies have examined associations between illness and graduation rates.¹⁶ Second, for those who earn diplomas, prospects for obtaining any further education and jobs offering advanced skills training are grim. The cumulative effects of low educational attainment, technical expertise, employment experience, and job availability severely handicap adults with SMI in vying for attractive jobs while markedly increasing their risks of remaining mired in unemployment and poverty.¹⁷ Thus, focusing integrated SE and mental health services toward employment in rural labor markets may help extract the most out of meager job opportunities and services.

To assist rural adults with SMI in accessing and benefiting from integrated vocational and mental health services oriented toward increasing chances of competitive labor market success, the South Carolina Department of Mental Health (SCDMH) and the Medical University of South Carolina (MUSC) collaborated to conduct 1 of the 8 site studies of the Employment Intervention Demonstration Program (EIDP).¹⁸ Each site selected and implemented its own SE and comparison programs and evaluated them with the EIDP's comprehensive cross-site assessment protocol. For the Sumter County catchment area of the SCDMH-operated Santee-Wateree Community Mental Health Center (SWCMHC), we designed and attempted to implement 2 empirically supported urban SE service models, Assertive Community Treatment With Integrated Vocational Rehabilitation (ACT-IVR)¹⁹ and Individual Placement and Support (IPS).²⁰ ACT-IVR tightly integrates vocational with mental health services within a self-contained provider team. IPS integrates its vocational services with its host agency's or another agency's mental health services.^{7, 21–22}

Our comparison program, a formal partnership between a local vocational rehabilitation agency and SWCMHC, provided parallel traditional vocational and mental health services, respectively. The rehabilitation agency's employment specialists prepared participants for competitive jobs by taking a gradual, stepwise approach that focused on developing skills to manage workplace demands in staff-supervised jobs set aside for adults with work disabilities.

In a 24-month randomized clinical trial, we addressed 5 specific questions: (1) What obstacles would a rural South Carolina setting pose to implementing and sustaining 2 evidence-based SE models (ACT-IVR and IPS) that integrate SE with specialty mental health services? (2) How do the competitive work outcomes of the ACT-IVR and IPS programs compare to a traditional program providing parallel services in a rural economy with a high proportion of extremely disadvantaged African American residents? (3) How are overall work outcomes affected by differing strategies for obtaining employment: (a) quickly pursuing work in the open labor market (ACT-IVR and IPS) versus (b) eliminating employment barriers by immediate placement into temporary set-aside jobs (traditional program)? (4) How do rates of income earned from competitive work change over time by program, a potential proxy for recovery of work capacities and community integration? and (5) secondarily, How do the programs' work outcomes compare after controlling for the number of participants' service contacts and accounting for the potential moderating effects of 7 preselected characteristics (job history, benefit status, diagnosis, education, race, age, and gender) commonly found in prior studies to be associated with work outcomes?²³

Methods

Geographic and Service Setting

From its main office in the Town of Sumter (population 39,000), SWCMHC serves 4 of the state's 46 counties, including the Sumter County study service area (population = 102,000; 665 square miles; density = 157/square mile). African American (47%) and white (49%) residents, neither of Hispanic/Latino origin, make up almost the entire population. In 1999, median county per capita income hovered near $16,000; 14% of persons aged 18 and older and 13% of families lived below the federal poverty level. About 62% of persons were in the labor force (56% for women); unemployment rates ranged from a high of 5.5% in 1997 to a low of 4.6% in 1998. Jobs clustered in economic sectors of retail trade (12%), manufacturing (24%), and educational, health, and social services (19%).

Selection Criteria, Recruitment, Enrollment, and Random Assignment

The EIDP Steering Committee defined the population for this study as meeting both the federal Center for Mental Health Services' criteria for severe and persistent mental illness, based upon diagnosis, illness duration, and level of disability,²⁴ and EIDP-specific criteria of age 18 or older, unemployed at study entry, and current and/or future interest in competitive employment.⁴ For our South Carolina site study, we added the criterion “SWCMHC client for at least 6 months” to reduce the probability of excessive attrition due to frequent relocation of residence typical of this rural South population. We intended these selection criteria to produce a sample representative of a broader population of adults with SMI and work disabilities, who were not currently in crisis and who expressed interested in working, versus a sample of adults with the most severe illnesses and disabilities typically enrolled by “Madison-model” ACT programs. The full-convened Institutional Review Boards of both SCDMH and MUSC approved the study plan in 1995 and amendments throughout the project period.

In 1996, SWCMHC estimated 1,600 to 1,800 of its 2,800 Sumter County Community Support Program (CSP) working-age clients would likely meet eligibility criteria. We asked SWCMHC providers to notify their clients of this study during routine service encounters. However, we do not know how many clients actually learned of the study during the 24-month recruitment window. SWCMHC providers furnished us with written documentation that 285 clients definitely requested study details. Using a brief screener of eligibility criteria, providers determined 20 as ineligible and judged another 37 as probably eligible but not interested. We later verified that all of these provider judgments of client eligibility were accurate. Research assistants (RAs) met individually with the remaining interested clients (n = 228) to obtain voluntary informed consent, describe study purposes and procedures, and reassess eligibility. All RAs were licensed or credentialed service providers or persons who had worked with the study population for at least 15 years. They determined 17 clients as ineligible; another 21 withdrew consent after their meetings. The remaining clients (n = 190) attended 1 RA-led research induction group meeting to further review study activities and address clients' concerns. After this meeting, 13 withdrew consent, yielding a final pool of 177 enrolled participants.

The random assignment protocol was generated and implemented by 2 MUSC research personnel otherwise unaffiliated with this study. A statistician prepared an SAS-generated restricted random assignment sequence (permuted blocks of 3) to balance time of study entry and sample size across study programs. He surrendered to a Charleston-based RA the assignment sequence, to which no investigator was permitted access. Sumter RAs transmitted enrolled participants' names to the Charleston RA, who assigned participants to programs.

Implementation Difficulties and Study Redesign

Our original study plan called for a 3-group trial comparing 2 newly implemented SE programs, ACT-IVR^{19, 25} and IPS,²⁰ to a traditional vocational rehabilitation program. We contracted with developers of the ACT-IVR and IPS models who, as independent external consultants, would train and mentor program staff and monitor program fidelity to ACT and IPS model standards throughout the 4-year project period.

Two months into the project period, SWCMHC had not yet recruited the requisite qualified providers to staff both the ACT-IVR and IPS programs, which is similar to recruitment problems described by other rural ACT programs.²⁶ The ACT-IVR and IPS consultants advised the investigators that simultaneous replication of these 2 model programs would not rapidly be achieved and recommended modifying the study plan. At this point, 7 participants had been assigned to each program. Between months 2 and 6 of the project period, the Center for Mental Health Services program official, the EIDP Steering Committee, the external consultants, and the investigator team explored alternative study designs while slowing participant accrual to the ACT-IVR and IPS programs to preserve low staff/participant ratios as stipulated by their respective model standards. During this deliberation period, 20, 27, and 31 participants were assigned to the ACT-IVR, IPS, and traditional programs, respectively, for a total of 27, 34, and 38.

We integrated the partially implemented and incompletely staffed ACT-IVR and IPS programs into a single ACT-IPS program, operating with ACT and IPS subteams composed of the ACT-IVR and IPS staff members, respectively. We eliminated the IPS condition and released its 34 participants from the project. The 27 ACT-IVR participants remained enrolled in the ACT-IPS program; the 38 traditional program participants continued without interruption. We assigned the subsequent 78 enrollees as pairs to the ACT-IPS and traditional programs, using the first 2 of the 3 elements in the random assignment sequence, giving a final participant tally of 66 (39 + 27) and 77 (39 + 38), respectively.

Sensitivity Analyses Assessing Threats of Internal Validity

Project redesign may have disrupted experiences for the 27 participants who began in the ACT-IVR program but transferred to the ACT-IPS program. Additionally, the study experiences of these 27 participants may have been discordant with those of the 39 subsequently assigned directly to the ACT-IPS program. To explore potential confounding induced by redesign, we specified a series of sensitivity analyses (available from the authors) comparing program and participant activity before and after redesign. The ACT-IPS program provided a modestly higher proportion of vocational services compared to the ACT-IVR program, but no substantive differences between ACT-IVR and ACT-IPS participants emerged across personal characteristics, types and amounts of services received, attrition rates, and work and other outcomes. However, such minor differences do not rule out that other potent but unobserved factors associated with the redesign account for observed variation in work outcomes.

Program Operations After Study Redesign

ACT-IPS Program

The ACT subteam consisted of a staff profile typical of rural ACT teams²⁷: 1 psychiatrist, 1 M.A.-level clinician, 2 B.A.-level registered nurses, and 2 B.A.-level case managers. The IPS subteam consisted of an M.A.-level social worker, who led the ACT-IPS program, and 2 B.A.-level employment specialists. An administrative assistant handled logistics for the entire program. About a third of the staff was African American, and the remainder, white. To tightly integrate vocational and mental health services, the subteams met daily together as a full program to allocate tasks to each IPS specialist and ACT staff member; formulate treatment plans emphasizing employment goals according to each participant's preferences, strengths, and weaknesses; select a mix of services tailored for each participant's work goals; and update each participant's single unified treatment record.

The ACT subteam operated as a “generalist” (i.e., interdisciplinary) team according to ACT model standards.²⁵ All staff provided all services, except vocational services, which became the responsibility of the IPS subteam, and specific services limited by licensure to specific professionals. They shared responsibility for every participant's welfare and sought to form strong working alliances between each participant and the entire ACT-IPS program. To minimize harm from acute illness episodes, the subteam maintained a very low staff/participant ratio (1:10) and was available on call 24 hours every day. No predetermined time limits were placed on service eligibility, regardless of relapse or treatment plan nonadherence.

The IPS subteam adhered to IPS model standards.²⁰ Employment specialists assessed each participant's past work experiences, current skills, and tolerance for type and intensity of job demands. Participants chose which jobs to pursue, although specialists strongly urged competitive jobs over work-adjustment experiences in protected settings. Together participants and specialists searched for competitive job openings and/or agreed to placement into new jobs “developed” jointly by specialists and local employers. Specialists provided time-unlimited support before, during, and after periods of employment. Like ACT staff, they shared responsibility for meeting every participant's employment goals.

Comparison Traditional Program

SWCMHC and the Genesis Center (GC), a local independent nonprofit vocational rehabilitation agency specializing in the employment of adults with SMI, signed a formal agreement creating the comparison program, which they named the Supported Employment Program (SEP), despite its traditional vocational rehabilitation philosophy. SWCMHC provided mental health and brokered case management services in parallel to GC's vocational services. To reduce differences in program service intensity between SEP and the larger ACT-IPS program, which might confound service intensity with work outcomes, SWCMHC dedicated 2 B.A.-level case managers to serve only SEP participants, at a staff/participant ratio not to exceed 1:30.

The GC program consisted of 10 full-time experienced employment specialists, ranging in education from doctoral degrees to high school diplomas. About equal numbers were African American and white. Subteams of specialists, each with a leader and shared caseload, met 2 to 4 times per week to review participants' progress and once per week with the executive director for supervision. All specialists developed participant work plans, conducted office-based work-readiness assessments, and provided on-the-job support at a staff/participant ratio at or below 1:10. GC operated 20 hours per day on weekdays and 4.5 hours on Saturday.

Although both GC and IPS viewed competitive work as an ultimate goal, IPS immediately aimed for competitive jobs, while GC introduced participants first to graduated work-adjustment experiences as preparation for handling competitive job demands. After assessing each participant's job skills and interests, GC specialists placed participants into 1 of its temporary, staff-supervised “set-aside” jobs, which differ from competitive jobs in several ways. First, GC contracted for these jobs directly with employers, vesting both with authority for terminating a job against a participant's wishes. In contrast, participants' contracts with employers in the competitive labor market, by law allows only the employer to terminate a job against a participant's wishes. Second, GC's agreements with employers stipulated that GC, not a participant, held ultimate responsibility for meeting employers' job performance standards. If a participant failed to meet standards, then either GC staff assigned another participant to the job or they worked the job until another participant became available to take the job. Third, GC set a 6-month time limit on holding set-aside jobs, after which the jobs turned over to new participants. For participants judged as ready for competitive work, specialists assisted with a brief job search and 2 weeks of job support. Any participant who failed to hold a competitive job could reenroll for additional GC work-adjustment services per protocol as above.

Fidelity of Implementation

ACT Subteam

On a quarterly basis, the external ACT consultants rated the ACT subteam's service performance using the 72-item ACT Fidelity Checklist (yes–no format) covering model standards organized into 13 domains (e.g., outreach and continuity of care, staff functioning as generalists, majority of services delivered in home and community settings).²⁵ Checklist psychometric properties have not yet been reported. Consultants reviewed service and job activity logs, treatment plans, and progress notes; interviewed staff and participants in office and field settings; and observed field interventions. Yearly averaged checklist scores rose steadily over the project period: 77, 80, 91, and 94% (% = number of yes answers to the 72 items), for years 1–4, respectively, indicating high fidelity in years 3 and 4. In reports and exit meetings with staff, consultants offered specific recommendations to improve service quality.

IPS Subteam

On a quarterly basis, the external IPS consultants rated the IPS subteam's service performance using the 15-item IPS Fidelity Scale.²⁸ Each item is rated on a 5-point Likert scale ranging from low (1) to high (5) adherence to each IPS model standard, with behavioral descriptions anchoring each point. Items are allocated to 3 rationally defined subscales: (a) staffing (e.g., specialists provide only vocational services), (b) organization (e.g., specialists integrate vocational with mental health services), and (c) services (e.g., rapid and individualized job search and placement). The range of total scale scores is 15 to 75 (alpha = .90); total scores clearly discriminate IPS and other SE programs from traditional vocational rehabilitation.²⁸ To rate fidelity, IPS consultants drew upon identical information sources as did the ACT consultants. Yearly averaged total scores rose steadily over the project period: 64, 69, 72, and 72, for years 1–4, respectively, indicating very high IPS model fidelity in years 2–4. In reports and exit meetings with staff, consultants proposed specific program changes to enhance service quality.

SEP

At the end of years 2 and 4, the first 2 authors assessed how closely SEP program services matched those specified in the original study plan. They examined SWCMHC and GC annual service and financial reports and interviewed staff and both executive directors. Staff provided services per study protocol with no significant deviations over the project period.

Measures

Employment Outcomes

Both IPS and GC employment specialists recorded attributes of all participants' jobs, including (a) duties; (b) who contracted for the job (participant or GC); (c) job status as set aside for persons with SMI or in the open labor market; (d) coworker disability profiles, ranging from all with disabilities to none; (e) methods used to obtain jobs (e.g., self, job development); (f) available benefits (e.g., insurance); and (g) reasonable accommodations for impairments. Specialists submitted weekly updates for each working participant, including hours worked, income earned, and reasons for job terminations. We classified a job as competitive if the employer (a) paid competitive wages above the federal minimum to participants and workers without disabilities alike, (b) did not set aside the job for adults with disabilities, and (c) located the job in a typical community setting and if (d) the participant contracted for the job.

Background Characteristics and Clinical and Psychosocial Outcomes

RAs conducted face-to-face interviews using the EIDP multisite protocol and assessed demographic characteristics, past psychiatric and substance misuse history, job experience, income sources, federal benefits, and other contextual information at baseline and at months 6, 12, 18, and 24. Participants received $25 for each interview completed. RAs evaluated psychiatric symptoms with the 30-item Positive and Negative Syndrome Scale (PANSS).²⁹ Each item is rated on a 7-point scale from not applicable (1) to extreme (7). We used White et al.'s “pentagonal model” containing 25 of the original 30 items, organized into 5 factor-analytically derived scales.³⁰ Internal consistency reliability estimates (alpha) for each scale averaged across the 5 interviews were positive (.71), negative (.79), autistic preoccupation (.60), activation (.70), and dysphoria (.69). Floor effects on all scales account for relatively low reliability estimates. RAs administered the brief version of the Quality of Life Interview (QOLI)³¹ concurrently with the PANSS. Participants rated their subjective satisfaction with housing, family relations, social relations, finances, and physical health on a 7-point Likert scale ranging from terrible (1) to delighted (7). Coefficient alphas for the 5 scales averaged across the 5 interviews ranged narrowly between .75 and .85.

Service Use

The multisite EIDP data-collection protocol defines 10 vocational and 7 mental health services (exact operational definitions can be retrieved from www.psych.uic.edu/EIDP/Service_Categories.pdf). ACT-IPS and SEP providers submitted daily service logs, noting, for each contact, the service type, need, duration, location, and all persons involved. For program comparisons, we grouped services into vocational and mental health service clusters.

Data Sources, Collection, and Quality Control

In this unblinded community-based implementation effectiveness study, we attempted to minimize data-collection bias and contamination by separating tasks among personnel and limiting their contact with each other. The investigator team trained each RA to administer the EIDP multisite interview protocol, supplemented by ongoing recalibration training provided by the EIDP Coordinating Center and the first author. On a continuing basis, our data entry specialist and data manager reviewed the accuracy of all providers' service delivery logs and all weekly logs of participant job activity compiled by employment specialists. Blind to study participant, the first author reviewed job logs for logical consistency. The data manager ran biweekly statistical logic and outlier checks to identify remaining problems. He contacted providers to verify data and correct errors and generated an audit trail of all database changes. Independent ACT and IPS consultants conducted fidelity assessments, and with the second author, they supervised and mentored ACT-IPS program providers. SEP supervisors from the SWCMHC and GC agencies monitored and mentored their staff.

Power Analysis

From data reported by the 2 SE randomized clinical trials completed by the time of study commencement,^32–33 we assumed that the average monthly competitive job income earned by their SE (M = $158, SD = $245) and comparison (M = $54, SD = $144) program participants would be reasonable targets for the ACT-IPS and SEP programs, respectively. Using t-tests with alpha set to .05 and power to .80, 59 participants per program were required to rule out sampling error as a plausible explanation for program differences in earned income.

Statistical Analysis Plan

Participant Characteristics, Services Received, and End Point Work Outcomes

We compared baseline characteristics of ACT-IPS and SEP participants, using chi-squared difference tests for nominal-scaled variables and Wilcoxon rank sum tests for ordinal-scaled variables. We profiled service receipt by cumulating total contacts across participants by program over the study period for each individual service and for the vocational and mental health service clusters. For work outcomes, we cumulated values across participants by program over the study period. Highly positively skewed distributions for both services and work outcomes required computation of medians (interquartile ranges) and comparison of program work outcomes with Wilcoxon rank sum tests and effect sizes computed from the U-statistic.³⁴

Income Earned Over Time

We divided the study period into six 4-month periods to generate enough measurement time points for testing the explanatory effects of time, program, and their interaction on income earned from competitive jobs. Across intervals, income appeared as highly positively skewed semicontinuous distributions with modes of zero. Percentages of zero values, positive values, and missing data points for the ACT-IPS program are 55.0, 33.6, and 11.4% and for the SEP program are 77.2, 11.5, and 11.3%.

Commonly used linear transformations fail to normalize these distributions to meet assumptions of parametric statistical models. For example, log transformation of earned income normalizes skewed nonzero values but leaves zero values unaffected. Conducting nonparametric analyses of these distributions by recoding values into categories offers no advantage either. For example, recoding income values into “yes–no” dichotomies for logistic regression analysis prevents addressing important questions about “how much income was earned at what rate over time.” Recoding income into ranks for Wilcoxon tests produces asymmetrical distributions dominated by “ties” at zero values. Thus, we treated income as a mix of 2 separate distributions of zero and positive values, representing 2 distinct but correlated “random processes.” These processes describe “whether or not a participant worked during each of the study's 6 intervals” and, conditional on working, “how much income a participant earned at what rate” over these intervals, respectively. The correlation between work probabilities and income amounts may yield information about the nature and pace of participants' recovery of work capacities and differential program impacts on work outcomes over time.

We evaluated these 2 random processes using mixed-effects mixed-distribution models with correlated random effects, which contain 2 components, 1 for each random process.³⁵ First, the “logistic” response component estimates the conditional probability of a zero value (working during a given interval or not) as a function of 3 explanatory fixed effects (time, program, and their interaction) and 1 between-participants random effect (intercept). Second, conditional on working, the “lognormal” response component estimates the mean of log-transformed positive values of income as a function of the same set of fixed effects specified for the logistic component, plus a between-participants random intercept and 1 within-participants random effect (residuals). We assumed random effects as bivariate normally distributed. A final random effect accounts for the correlation (i.e., link) between the 2 components' random intercepts, which yields the strength of association between work probability and income amount per interval.

We conducted these analyses using a SAS macro, “MIXCORR,”³⁶ which sequentially calls in SAS PROC GENMOD and SAS PROC NLMIXED procedures to estimate, by interval, work probability (logistic component) and, conditional on working, log-transformed average income (lognormal component), respectively. We centered time on the final interval to assess the full 24-month program main effect on income. Using a model building strategy, we evaluated nested models of progressing complexity. Model 1, “unconditional means,” contains only 1 fixed effect (intercept) and serves as the baseline upon which to compare how much the 3 explanatory fixed effects account for work probabilities and income amounts. Adding a fixed linear time effect to both components generates Model 2, “unconditional growth.” Introducing fixed effects for study program (ACT-IPS and SEP) and their interaction with time to Model 2 yields Model 3, “full program/time model (PTM).”

In exploratory analyses (Models 4–5), we re-estimated the PTM explanatory effects on income, controlling for time-varying effects of clustered vocational and mental health service contacts by interval in separate analyses. Although we do know that the 2 programs functioned very differently in day-to-day practice, we concede that the nonspecific effects of simple receipt of any type or amount of service might more parsimoniously account for the effects of a program (and its interaction with time) on earned income.

In a final set of exploratory analyses (Models 6–11), we re-estimated the PTM explanatory effects on income, specifying 6 time-invariant demographic and diagnostic characteristics as main effects and interactions with program (i.e., potential moderators of program impact) in separate analyses. We selected these characteristics (job history, benefit status, diagnosis, education, race, age, and gender) in advance owing to their common association with work outcomes reported in prior studies.²³

Secondary Psychiatric Symptom and Quality of Life Analyses

Both PANSS factor-derived scale scores and self-reported QOLI scale scores were normally distributed. We conducted linear mixed-effect regression models (SAS PROC MIXED) for each scale score, specifying the same set of explanatory fixed effects as for the MIXCORR models, and also specified random effects for final status (intercept) and time. Because 5 interviews were conducted at baseline and at 6-month intervals, time was defined as four 6-month intervals and centered on the final interval, and baseline scores on each scale were specified as covariates.

Results

Characteristics of Participants at Baseline and Attrition Patterns

At baseline, characteristics of both programs' participants were similar (table 1). At least 80% of both programs' participants completed the comprehensive assessment interview at months 6, 12, 18, and 24, with 1 exception (74% of ACT-IPS participants at month 12). Figure 1 summarizes participants' study-related activity from random assignment through the 24-month end point. Enrollment began in June 1996 and ended in May 1998; follow-up concluded in May 2000. Immediately after program assignment, 3 SEP and 5 ACT-IPS participants withdrew. A similar and high percentage of ACT-IPS and SEP participants received vocational services through the first 6 months (86 versus 83%). Over time, however, a much higher percentage of ACT-IPS participants received vocational services (82 versus 56% at 12 months and 67 versus 26% at end point). A similar and high percentage of ACT-IPS and SEP participants received mental health services through 18 months (82 versus 78%), with modest divergence by study end point (79 versus 69%). Overall, both programs demonstrated steady rates of attrition across the study period, defined as the final month in which a participant received any service and/or worked as reported directly by study providers. The ACT-IPS and SEP programs lost about 7 and 2–3% of their participants per 6-month interval, respectively. Most ACT-IPS and SEP participants remained active through 18 months (83 versus 79%), but a modestly higher percentage of ACT-IPS participants completed the study (82 versus 70%). Voluntary withdrawal accounted for most attrition, followed by relocation outside the service area.

Table 1.

Baseline Characteristics of Study Participants by Program

Characteristic	Supported Employment Program (n = 77)	Assertive Community Treatment–Individual Placement and Support Program (n = 66)	Statistica	P Value
Age			U = 2,811, z = 1.03	.29
18–25 years	9 (11.7)	9 (13.6)
26–45 years	59 (76.6)	42 (63.6)
>46 years	9 (11.7)	15 (22.7)
Female	42 (54.6)	47 (71.2)	χ2(1) = 4.20	.06
Race			χ2(2) = 0.10	.95
African American, not Hispanic	60 (77.9)	50 (75.8)
White, not Hispanic	14 (18.2)	13 (19.7)
Other	3 (3.9)	3 (4.6)
Not married/not cohabitating	63 (81.8)	55 (83.3)	χ2(1) = 0.06	.83
Educational attainment			U = 2,707, z = 0.44	.67
Some high school	38 (49.4)	31 (47.0)
High school/general equivalency diploma	19 (24.7)	15 (22.7)
Some college/technical school	20 (26.0)	20 (30.3)
Monthly income			U = 2,275, z = −1.51	.15
$0–500	37 (48.1)	42 (63.6)
$500–750	32 (41.6)	16 (24.2)
>$750	8 (10.4)	8 (12.1)
Mental illness			χ2(1) = 2.34	.15
Schizophrenia spectrum	57 (74.0)	41 (62.1)
Mood spectrum	20 (26.0)	25 (37.9)
Positive and Negative Syndrome Scale, psychiatric symptoms (mean [SD])
Positive (4 items)	7.9 (3.2)	8.3 (3.4)	t(141) = −0.72	.47
Negative (7 items)	17.0 (6.0)	16.9 (5.1)	t(141) = 0.08	.93
Autistic preoccupation (5 items)	11.2 (3.8)	11.4 (4.0)	t(141) = −0.24	.81
Activation (4 items)	7.1 (2.6)	7.6 (3.3)	t(141) = −1.10	.27
Dysphoria (5 items)	12.5 (4.6)	13.1 (5.2)	t(141) = −0.67	.51
Alcohol abuse/dependence (current)	6 (7.8)	7 (10.6)	χ2(1) = 0.34	.58
Drug abuse/dependence (current)	6 (7.8)	5 (7.6)	χ2(1) = 0.00	.99
Quality of Life Interview, self-reported satisfaction (mean [SD])
Finances (3 items)	9.5 (4.8)	9.1 (5.2)	t(141) = 0.48	.63
Health (3 items)	13.4 (4.8)	13.8 (4.7)	t(141) = −0.51	.61
Housing (3 items)	13.4 (4.6)	13.5 (5.2)	t(141) = −0.11	.91
Social life (3 items)	15.4 (3.4)	15.8 (3.7)	t(141) = −0.69	.49
Family (2 items)	9.4 (3.2)	9.0 (3.8)	t(141) = 0.70	.48
Paid work past 5 yearsb			U = 2,188, z = 0.25	.77
>0–6 months	30 (40.0)	21 (33.9)
6–12 months	21 (28.0)	23 (37.1)
>12 months	24 (32.0)	18 (29.0)
Duration of most recent jobc			U = 1,490, z = −1.15	.27
<1–4 months	28 (39.4)	28 (47.5)
4–12 months	21 (29.6)	18 (30.5)
>12 months	22 (31.0)	13 (22.0)
Consecutive months unemployed prior to study entryd			U = 1,097, z = 0.54	.61
0 months	6 (9.7)	3 (5.6)
1–12 months	17 (27.4)	15 (27.8)
>12 months	39 (62.9)	36 (66.7)
Supplemental Security Income beneficiary	34 (44.2)	23 (34.8)	χ2(1) = 1.28	.30
Social Security Disability Income beneficiary	26 (33.8)	17 (25.8)	χ2(1) = 1.08	.36
Social Security Disability Income or Supplemental Security Income beneficiary	50 (64.9)	37 (56.1)	χ2(1) = 1.17	.31

Note: Data are given as number (percentage) of participants, except where indicated. Numbers vary due to missing data.

^aFisher's exact test used for chi-squared tests; exact test used for Wilcoxon rank sum tests.

^bData missing for 2 SEP and 4 ACT-IPS participants.

^cConditional on past history of working; data missing for 3 SEP and 2 ACT-IPS participants.

^dData missing for 15 SEP and 12 ACT-IPS participants.

Fig. 1.

Participant Flow—Random Assignment Through End of 24-Month Participation Period.

Note: Numbers (percentages) for all variables refer to service receipt and job activity within each of the 4 6-month intervals. “Active sample remaining” (i.e., the inverse of attrition) is defined as the percentage of participants in each program with any form of continuing study involvement through each succeeding 6-month interval (i.e., receipt of vocational or mental health services or employed as reported directly by study providers). SEP = Supported Employment Program, ACT-IPS = Assertive Community Treatment–Individual Placement and Support Program, IPS = Individual Placement and Support.

Program Service Profiles and Service Integration

Vocational and Mental Health Services

The programs' vocational service profiles differed considerably in emphasis and diversity (table 2). More than two-thirds of ACT-IPS participants received 7 of the 10 EIDP-defined vocational services; over half received 9 of 10. However, only a simple majority of SEP participants received 2 of 10. Cumulated over the study period, ACT-IPS participants logged many more vocational service contacts (medians [Mdns] = 54 versus 9) distributed across far more study months (57 versus 22%). The programs' mental health service profiles differed less compared to vocational services. Over two-thirds of ACT-IPS participants received 4 of the 8 EIDP-defined mental health services; about half received 6 of 8 (table 2). More than two-thirds of SEP participants received 3 of 8, but few received other mental health services. Cumulated over the study period, ACT-IPS participants logged more mental health service contacts (Mdns = 86 versus 59) distributed across slightly more study months (78 versus 67%), and their grand total of service contacts far exceeds those of SEP participants (Mdns = 140 versus 83) distributed across slightly more study months (83 versus 73%). No “contamination” of program service receipt was evident: no SEP participants received services from the ACT-IPS program; no ACT-IPS participants received services from SEP case managers, but on rare occasions, they received SWCMHC mental health services when ACT-IPS providers were unavailable. Only 5 ACT-IPS participants requested a placement into a GC set-aside job.

Table 2.

Service Profile for Each Program

	Supported Employment Program (n = 77)			Assertive Community Treatment–Individual Placement and Support Program (n = 66)
Service	n (%)	Median	Interquartile Range	n (%)	Median	Interquartile Range
Vocational services
Vocational assessment	13 (17)	0	0–0	59 (89)	3	1–4
Job development or finding	11 (14)	0	0–0	60 (91)	5.5	2–12
Collaboration—employers	28 (36)	0	0–1	46 (70)	3	0–5
Vocational support groups	0 (0)	0	0–0	14 (21)	0	0–0
Collaborate—family/friends	13 (17)	0	0–0	34 (52)	1	0–2
Vocational treatment planning	63 (82)	2	1–4	57 (86)	5	2–11
Skills training	37 (48)	0	0–3	50 (76)	3	1–7
Vocational counseling	12 (16)	0	0–0	56 (85)	6	2–14
Job support—on-site	26 (34)	0	0–14	35 (53)	1	0–2
Transportation	20 (26)	0	0–1	56 (85)	10	3–21
All vocational service contacts	69 (90)	9	2–40	61 (92)	54	26–71
Mental health services
Case management	74 (96)	26	16–43	61 (92)	44	29–84
Family/couples counseling	0 (0)	0	0–0	6 (9)	0	0–0
Emergency care	19 (25)	0	0–0	40 (61)	1	0–4
Psychiatric evaluation	60 (78)	2	1–5	60 (91)	8	4–11
Individual counseling	33 (43)	0	0–2	46 (70)	2	0–5
Group counseling	13 (17)	0	0–0	33 (50)	1	0–2
Medication management	69 (90)	15	6–28	60 (91)	22	11–36
Partial hospitalization	2 (3)	0	0–0	1 (2)	0	0–0
All mental health service contacts	74 (96)	59	31–87	61 (92)	86	64–157
All service contacts	74 (96)	83	40–134	61 (92)	140	96–230

Note: Number (percentage) indicates participants who received services.

Integration of Vocational With Mental Health Services

IPS employment specialists coordinated vocational assessment, planning, counseling, and job development/finding with ACT interventions reducing psychiatric symptoms impairing participants' job performance. To maximize the time and energy that participants could devote to work, ACT staff quickly addressed pressing housing, medical, and other basic needs. When necessary, IPS specialists arranged travel to work sites. With participants' permission, specialists enlisted their natural support networks (e.g., families and friends) and employers to aid vocational development. Over time, the ACT-IPS program adjusted services to match participants' evolving job aspirations, enhance acquired job skills, and build enthusiasm to pursue new jobs after loss of a job.

Operating from a more narrow vocational development approach, GC employment specialists prepared participants for competitive work by placing them in closely supervised set-aside jobs and concentrating on vocational planning and job skill development. In parallel, SEP case managers brokered mental health and support services, while SEP clinicians evaluated psychiatric status and managed psychotropic medications. Numerous obstacles prevented SEP providers from coordinating vocational with mental health services: they worked out of various locations on different schedules, reported to different supervisors, dealt with differing reimbursement policies, and provided services based on different therapeutic orientations.

Job Outcomes: End Point Analyses

Employment Rates and Income Earned

We report outcomes for both competitive and all jobs for 2 reasons. First, most constituencies agree that, for persons with SMI and limited work histories, holding competitive jobs may increase chances for recovering valued adult roles and increasing community involvement.^37–38 Second, advancing toward economic self-sufficiency depends upon earning income, regardless of job type. Of the 48 ACT-IPS working participants, 42 (87%) obtained competitive jobs. In contrast, of the 51 SEP working participants, only 20 (39%) held competitive jobs. The first jobs landed by 9 of these 20 were competitive jobs they obtained on their own. Only 11 of these 20 transitioned from an SEP set-aside job into a competitive job. Overall, over twice the percentage of ACT-IPS participants obtained a competitive job (63.6 versus 26.0%; p < .001, effect size [ES] = 0.38), but differences for all jobs were less (72.7 versus 66.2%; p = .40, ES = 0.07; table 3). ACT-IPS participants earned much more income from competitive jobs (Mdn = $549, interquartile range [IQR] = $0–5,145, versus Mdn = $0, IQR = $0–$40; p < .001, ES = 0.70), but differences for all jobs were less (Mdn = $1,337, IQR = $0–5,495, versus Mdn = $743, IQR = $0–2,800; p = .07, ES = 0.59).

Table 3.

Cumulated Employment Outcomes by Program

	Competitive Jobs									All Jobs
	Supported Employment Program (n = 77)			Assertive Community Treatment—Individual Placement and Support Program (n = 66)						Supported Employment Program (n = 77)			Assertive Community Treatment—Individual Placement and Support Program (n = 66)
Outcome	n	N	%	n	N	%	Statistic	P Value	Effect Size	n	N	%	n	N	%	Statistic	P Value	Effect Size
N who worked	20	77	26.0	42	66	63.6	χ2(1) = 20.5	<.001	0.38	51	77	66.2	48	66	72.7	χ2(1) = 0.71	.40	0.07
Jobs by program	35	124	28.0	89	124	71.2		–	–	92	203	45.3	111	203	54.7		–	–

	Competitive Jobs							All Jobs
	Supported Employment Program (n = 77)		Assertive Community Treatment—Individual Placement and Support Program (n = 66)					Supported Employment Program (n = 77)		Assertive Community Treatment—Individual Placement and Support Program (n = 66)
Outcome	Median	Interquartile Range	Median	Interquartile Range	Statistic	P Value	Effect Size	Median	Interquartile Range	Median	Interquartile Range	Statistic	P Value	Effect Size
Jobs per worker	2	(1–2)	2	(1–3)	–	–	–	1	(1–2)	2	(1–3)	–	–	–
Total earned income
All	0	(0–40)	549	(0–5,145)	U = 3,551, z = 4.52	.001	0.70	743	(0–2,800)	1,337	(0–5,495)	U = 2,983, z = 1.81	.07	0.59
With jobs	1,857	(747–4,443)	2,855	(900–6,968)	U = 327, z = 0.79	.43	0.39	1,954	(743–4,291)	3,388	(1,068–8,567)	U = 1,501, z = 1.93	.06	0.61
Total weeks worked
All	0	(0–1)	8.5	(0–41)	U = 3,541, z = 4.47	.001	0.70	10	(0–43)	16	(0–51)	U = 2,804, z = 1.08	.28	0.55
With jobs	22.5	(12–44.5)	31.5	(9–53)	U = 316, z = 0.64	.52	0.38	25	(10–56)	33.5	(14.5–65.5)	U = 1,322, z = 0.68	.50	0.54
Total hours worked
All	0	(0–8)	108	(0–800)	U = 3,549, z = 4.51	.001	0.70	171	(0–601)	296	(0–1032)	U = 2,908, z = 1.51	.13	0.57
With jobs	360	(146–844)	485	(139–1139)	U = 325, z = 0.77	.44	0.39	434	(171–914)	602	(212–1356)	U = 1,426, z = 1.41	.16	0.58
Weeks/joba	11.5	(7.8–23.5)	12.5	(6.8–22.2)	U = 261, z = 0.17	.86	0.31	14	(9–27)	13	(7–27)	U = 1,116, z = 0.75	.46	0.46
Weeks/longest joba	20	(10–31)	19	(8–39)	U = 420, z = 0.00	.99	0.50	21	(10–40)	19	(9–41)	U = 1,201, z = 0.16	.88	0.49
Weeks to first joba	46	(31–70)	19	(7–39)	U = 616, z = 2.95	.005	0.73	4.4	(0–32.7)	19	(7– 39)	U = 1,638, z = 2.90	.005	0.67
Hours/week/joba	20	(9–27)	22	(14–33)	U = 343, z = 1.15	.25	0.41	16	(11–20)	21	(14–32)	U = 1,593, z = 2.58	.005	0.65
Wage/job ($/h)a	5.50	(5.13–5.60)	5.33	(5.03–6.00)	U = 410, z = 0.14	.89	0.49	4.75	(3.56–5.19)	5.15	(4.83–5.56)	U = 1,742, z = 3.62	<.001	0.71

Note: Effect sizes for chi-squared tests are phi coefficients; effect sizes for Wilcoxon rank sum tests derive from the U-statistic.³⁴ “All” indicates all participants randomly allocated to each of the 2 programs; “with jobs” indicates participants who worked during the 24-month study period.

^aConditional on working.

Two related factors account for the decrease in effect sizes for the intent-to-treat sample on income earned from competitive jobs (0.70) versus all jobs (0.59). First, 80% (89/111) of jobs held by the 48 working ACT-IPS participants were competitive, twice that of 38% (35/89) held by the 51 working SEP participants (table 3). Thus, ACT-IPS participants earned a higher proportion of their total income from competitive jobs, whereas SEP participants earned a higher proportion from noncompetitive jobs. Second, variations of earned income (interquartile ranges) between ACT-IPS and SEP participants were much higher for competitive jobs (IQRs = $0–5,145 versus $0–40) compared to all jobs (IQRs = $0–5,495 versus $0–2,800), despite the fact that their respective medians increased about equally (Mdns = $549–1,337 versus $0–743).

Weeks and Hours Worked

At competitive jobs, ACT-IPS participants worked more weeks and hours, but program differences decreased for all jobs by similar magnitudes as those for income (table 3). Conditional on working at competitive jobs, ACT-IPS participants logged similar weeks per job, weeks per longest job, and hours per week per job at about equal wage rates (table 3). They obtained their first competitive jobs much earlier in the study period (Mdns = 19 versus 46 weeks; p = .005, ES = 0.73) but their first paid jobs of any kind much later (Mdns = 19 versus 4.4 weeks; p = .005, ES = 0.67), primarily due to immediate placement of SEP participants into GC set-aside jobs. Small to moderate effect sizes (<0.60) for most comparisons of outcomes that were defined as conditional on working failed to reach statistical significance due to lack of statistical power resulting from smaller sample sizes.

Job Features

Compared to GC specialists, IPS specialists placed relatively more of their participants into competitive jobs developed jointly with local employers (16 versus 3%) and helped participants find relatively more existing job openings (29 versus 6%). Relatively fewer ACT-IPS compared to SEP participants obtained competitive jobs on their own (38 versus 60%) and through informal contacts (17 versus 31%). Overall, the ACT-IPS program aided a much higher proportion of its participants in getting competitive jobs (45 versus 9%), reflecting the IPS assertive, long-term approach to competitive work.

Very few competitive jobs came with benefits: less than 1 in 5 jobs offered any insurance; only 1 in 4 granted any paid leave. Most job functions, defined by the Dictionary of Occupational Titles (DOT),³⁹ required few technical skills. Competitive jobs were equally distributed across 3 of 9 DOT occupational groups: “service” (e.g., housekeeping), “processing” (e.g., poultry products), and “structural work” (e.g., home construction).

Job Terminations

Almost half of both programs' participants quit competitive jobs before securing a new job; employers laid off or fired about a third. IPS and GC employment specialists determined reasons for job losses from discussions with participants and, when they granted permission, employers and natural support networks. Reasons for job losses are reported for the job as the unit of analysis; multiple reasons were cited for about 25% of jobs. Because 1 SEP participant and 12 ACT-IPS participants continued working in competitive jobs at study end point, we report on 34 of 35 SEP jobs and 77 of 85 ACT-IPS jobs. Loss of ACT-IPS and SEP competitive jobs, respectively, due to quitting or firing, were primarily associated with symptoms impairing job performance (11.7 and 8.8%), job pressures (11.7 and 2.9%), inability to perform job tasks (13.0 and 5.9%), dissatisfaction with job duties (9.1 and 5.9%), and discontinuation of job positions (9.1 and 23.5%). Very few quit jobs out of dissatisfaction with schedules, hours, pay rate, or experiences of discrimination. Few were fired due to conflicts with coworkers and/or supervisors. Surprisingly, only 1 of 31 workers receiving federal benefits quit a competitive job from fear that earning income would lead to benefits termination.

Job Outcomes: Time Trends of Income Earned From Competitive Jobs

Descriptive Statistics

Percentages of ACT-IPS participants working in competitive jobs by 4-month intervals ranged narrowly between 33 and 38% throughout the study period. In contrast, much lower percentages of SEP participants worked by interval, ranging from less than 10% during the first 3 intervals, reflecting their early involvement in GC set-aside jobs, to a high near 20% in the final interval. Conditional on working, ACT-IPS participants earned more income per interval throughout the study period compared to SEP participants, although differences varied inconsistently over time, between small in the second interval (Mdn = $1,099, IQR = $670–2,635, versus Mdn = $930, IQR = $190–1,140) and moderately large in the fifth interval (Mdn = $2,084, IQR = $1,156–3,455, versus Mdn = $1,284, IQR = $300–1,972).

Primary Regression Models

We evaluated the effects of the 3 explanatory variables—program, time, and their interaction—on the probability of working and the amount of income earned per interval, with a sequence of nested mixed-effects, mixed-distribution regression models. Model 1, the unconditional means model, yields, for all participants regardless of program and time, the predicted average probability of working a competitive job (logistic intercept = −2.35, p < .001) and the average amount of income earned (lognormal intercept = 5.58, p < .001; table 4). Model 1's parameter estimates and fit indexes provide a baseline to judge the impact of the 3 explanatory variables on income earned over time. Model 2, the unconditional growth model, shows, for all study participants, a considerable increase in the probability of working over time (logistic time = 0.32, p < .001) but only a slight increase in earned income over time (lognormal time = 0.14, p < .001; table 4). The explanatory effect of time markedly improves model fit.

Table 4.

Work Probabilities and Income Earned From Competitive Jobs Over Time by Program

	Model 1: Unconditional Means		Model 2: Unconditional Growth		Model 3: Program × Time		Model 4: Vocational Servicesa		Model 5: Mental Health Servicesa
Parameter	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value
Probability of working (logistic component)
Intercept	−2.35 (0.33)	<.001	−1.65 (0.37)	<.001	−2.13 (0.52)	<.001	−2.31 (0.55)	<.001	−2.34 (0.56)	<.001
Time			0.32 (0.07)	<.001	0.74 (0.14)	<.001	0.80 (0.15)	<.001	0.77 (0.14)	<.001
Program (1 = ACT-IPS, 0 = SEP)					1.31 (0.65)	.046	1.29 (0.67)	.057	1.28 (0.67)	.06
Program × Time interaction					−0.63 (0.17)	<.001	−0.63 (0.17)	<.001	−0.62 (0.17)	<.001
Covariateb							0.031 (0.013)	.015	0.016 (0.010)	.11
Random intercept	6.71 (1.74)	<.001	7.52 (1.95)	<.001	6.35 (1.67)	<.001	6.88 (1.88)	<.001	6.82 (1.84)	<.001
Income from competitive job (lognormal component)
Intercept	5.58 (0.25)	<.001	5.86 (0.26)	<.001	5.30 (0.39)	<.001	5.32 (0.39)	<.001	5.23 (0.40)	<.001
Time			0.14 (0.04)	.001	0.15 (0.10)	.11	0.14 (0.10)	.15	0.16 (0.10)	.10
Program (1 = ACT-IPS, 0 = SEP)					1.10 (0.38)	.005	1.12 (0.38)	.003	1.11 (0.39)	.005
Program × time interaction					−0.009 (0.11)	.93	−0.013 (0.11)	.90	−0.002 (0.11)	.99
Covariateb							−0.006 (0.01)	.39	0.004 (0.005)	.51
Residual (within participants)	0.79 (0.10)	<.001	0.72 (0.09)	<.001	0.72 (0.09)	<.001	0.72 (0.09)	<.001	0.71 (0.09)	<.001
Random intercept	1.47 (0.48)	.002	1.56 (0.49)	.002	1.38 (0.43)	.002	1.33 (0.43)	.002	1.44 (0.46)	.002
−2LL/AIC	1,200	1,215	1,168	1,184	1,128	1,152	1,121	1,149	1,126	1,154

	Selected Models (7–12)
	Job Months in Past 5 Yearsc		Supplemental Security Income/Social Security Disability Incomed		Diagnosise		Education Levelc		Racef
Parameter	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value	Est (SE)	P Value
Probability of working (logistic component)
Intercept	−2.41 (0.55)	<.001	−0.70 (0.79)	.38	−2.77 (0.62)	<.001	−1.98 (0.53)	<.001	−2.52 (0.58)	<.001
Time	0.72 (0.14)	<.001	0.74 (0.14)	<.001	0.74 (0.14)	<.001	0.74 (0.14)	<.001	0.74 (0.14)	<.001
Program (1 = ACT-IPS, 0 = SEP)	1.08 (0.67)	.11	0.76 (1.05)	.47	1.60 (0.77)	.039	1.23 (0.66)	.06	1.48 (0.71)	.040
Program × time interaction	−0.61 (0.16)	<.001	−0.63 (0.17)	<.001	−0.63 (0.17)	<.001	−0.64 (0.17)	<.001	−0.63 (0.17)	<.001
Covariate	0.051 (0.022)	.025	−1.87 (0.87)	.033	2.25 (0.86)	.010	0.35 (0.33)	.28	2.02 (0.95)	.034
Covariate × program interaction	0.014 (0.029)	.63	0.69 (1.16)	.55	−1.24 (1.11)	.27	−0.13 (0.41)	.75	−0.77 (1.28)	.55
Random intercept	4.88 (1.33)	<.001	5.86 (1.56)	<.001	5.80 (1.54)	<.001	6.23 (1.64)	<.001	5.95 (1.58)	<.001
Income from competitive job (lognormal component)
Intercept	5.26 (0.41)	<.001	6.44 (0.44)	<.001	4.69 (0.45)	<.001	5.43 (0.38)	<.001	5.25 (0.43)	<.001
Time	0.16 (0.10)	.10	0.17 (0.09)	.07	0.18 (0.09)	.06	0.18 (0.09)	.06	0.15 (0.10)	.13
Program (1 = ACT-IPS, 0 = SEP)	0.92 (0.41)	.026	0.49 (0.51)	.34	1.53 (0.46)	.001	1.00 (0.38)	.010	1.03 (0.43)	.017
Program × time interaction	−0.018 (0.11)	.86	−0.03 (0.11)	.78	−0.03 (0.11)	.74	−0.034 (0.11)	.74	−0.004 (0.11)	.97
Covariate	0.022 (0.013)	.097	−1.45 (0.45)	.002	1.66 (0.49)	.001	0.46 (0.22)	.036	0.55 (0.53)	.30
Covariate × program interaction	0.006 (0.016)	.71	0.77 (0.56)	.17	−1.18 (0.60)	.051	−0.25 (0.25)	.31	0.10 (0.67)	.88
Residual (within participants)	0.72 (0.09)	<.001	0.74 (0.10)	<.001	0.72 (0.09)	<.001	0.70 (0.09)	<.001	0.72 (0.09)	<.001
Random intercept	1.05 (0.34)	.002	1.02 (0.34)	.004	1.24 (0.39)	.002	1.37 (0.42)	.002	1.25 (0.40)	.002
−2LL/AIC	1,111	1,143	1,115	1,147	1,114	1,146	1,121	1,153	1,121	1,153

Note: SEP = Supported Employment Program, ACT-IPS = Assertive Community Treatment–Individual Placement and Support Program, Est = estimate, SE = standard error, –2LL = minus twice the LogLikelihood, AIC = Akaike Information Criterion. Time is defined as 6 4-month intervals and centered on the last interval. The sample size is 77 for SEP and 66 for ACT-IPS. Smaller values for –2LL and AIC indicate better model fit.

^aNumber of contacts per interval.

^bCovariate = vocational and mental health services and demographic and diagnostic characteristics entered into Model 3 (program × time).

^cCentered at each program's median.

^dSupplemental Security Income/Social Security Disability Income: 1 = yes, 0 = no.

^eDiagnosis: 1 = mood spectrum, 0 = schizophrenia spectrum.

^fRace: 1 = Caucasian, 0 = African American.

Model 3, the “full program–time model,” reveals that, although ACT-IPS participants were much more likely to work during the final interval (logistic program = 1.31, p < .001), differences in work probabilities between programs decreased markedly from the first to the last interval (logistic program × time interaction = −0.63, p = .001; table 4). Conditional on working, ACT-IPS participants earned modestly more income (lognormal program = 1.10, p = .005), but the relative difference between programs did not change over time (lognormal interaction = −0.01, p = .93). Statistically significant random intercepts for both logistic and lognormal components (6.35 and 1.38, respectively; p < .001) indicate that explanatory fixed effects for time, program, and their interaction leave “unexplained” much between-participant variation in work probabilities and income earned over time, despite improved model fit.

The high positive correlation between the PTM's logistic and lognormal random intercepts (r = 0.92, p < .001) indicates that, as the percentage of ACT-IPS and SEP participants working per interval increased over time, on average, they earned more income per interval over time at about the same rate. Were this correlation negative, then a decreasing percentage of participants would have worked per interval over time but would have, however, earned more income per interval over time. This would be a good outcome for a few participants but a poor program outcome.

Secondary Regression Models

For vocational service contacts, both ACT-IPS and SEP participants logged the most in the first interval (Mdn = 13, IQR = 5–22, versus Mdn = 2, IQR = 0–11) and the least in the final interval (Mdn = 5, IQR = 0–9, versus Mdn = 0, IQR = 0–1). For mental health service contacts, both ACT-IPS and SEP participants logged the most in the first interval (Mdn = 44, IQR = 22–62, versus Mdn = 15, IQR = 8–22) and the least in the final interval (Mdn = 10, IQR = 2–10, versus Mdn = 4, IQR = 0–10). Controlling for the time-varying effects of vocational and mental health service contacts did not substantively alter the magnitude or direction of the PTM's explanatory effects on work probabilities and income earned over time (table 4).

Four of 6 participant characteristics were substantively associated with work probability; 3 of 6 were substantively associated with income earned. However, a lack of statistically significant interactions with program rules out these characteristics as moderators of program impact in this study sample and setting (table 4). The characteristics associated with a lower likelihood of working during the study period are race (African American), diagnosis (schizophrenia), work in the past 5 years (fewer months), and federal benefits (currently on the rolls). Conditional on working, the characteristics associated with less income earned during the study period are diagnosis (schizophrenia), educational attainment (less), and federal benefits (currently on the rolls). For all jobs, a parallel set of identically specified models yielded similar program patterns of work probabilities and income earned over time but smaller program differences compared to competitive jobs (results not shown but available from the authors).

Secondary Symptom and Quality of Life Outcomes

We found no substantive differences between both programs' participants and no substantive change from baseline over time for psychiatric symptoms (PANSS scale scores) and self-reported quality of life (QOLI scale scores; results not shown but available from the authors). Floor effects (i.e., less than mild symptoms) probably account for null findings across all PANSS scale scores. The low prevalence of substance use disorders (<10%) at baseline did not change over time. Adverse events included psychiatric hospitalization (SEP: n = 21, 27%; ACT-IPS: n = 17, 26%), a period of homelessness for 1 SEP and 1 ACT-IPS participant, and the sentencing of 3 SEP and 3 ACT-IPS participants to prison. A causal relationship between these events and study participation could not be demonstrated.

Discussion

Summary of Implementation Problems

We addressed 5 questions in this study. First, in a rural South Carolina county, we encountered formidable difficulties with implementing evidence-based model programs that integrate vocational rehabilitation with specialty mental health services. Recruiting and retaining the necessary numbers of qualified providers to staff the new ACT-IVR and IPS programs proved to be insurmountable. However, merging these 2 partially implemented programs formed the fully staffed ACT-IPS program, whose work outcomes for rural adults with SMI and limited employment experience compared favorably to those of urban-based SE programs. Maintaining ACT-IPS program effectiveness throughout the project period required ongoing training, mentoring, and fidelity monitoring. Upon termination of federal grant support, the South Carolina Department of Mental Health picked up the funding to sustain the ACT-IPS program into the present.

Summary of Competitive Work Outcomes

Second, the high participation rates through study end point and lack of marked differential participant attrition by program strengthen the validity of program comparisons. Across a set of competitive work outcomes, the ACT-IPS integrated service program outperformed the SEP parallel service program. Compared with SEP participants over the 24-month study period, more than twice the percentage of ACT-IPS participants held competitive jobs (64 versus 26%). Conditional on working in competitive jobs, they earned about 1.5 times the median income ($2,855 versus $1,857) and worked just under 1.5 times the median hours and weeks. Employment-focused integrated services are superior for enhancing job performance and marketability compared to parallel services, a straightforward inference consistent with findings from 2 recent SE process analyses reporting strong positive associations between service integration and competitive work outcomes.^40–41 Moreover, providing integrated services in participants' home and community settings probably facilitates productive long-term program–participant collaborations, by overcoming logistical and geographic impediments to accessing a full range of rural-based services.^{12, 17}

Effectiveness of Different Strategies for Obtaining Competitive Work

Third, although the SEP program eliminated barriers to work by placing participants immediately into its time-limited set-aside jobs and reenrolled any participant subsequently unable to hold a competitive job, the program did not increase its participants' competitive work activity over the study period. Of the 51 SEP participants who worked, only 20 held competitive jobs. Only 11 of these 20 first worked in a set-aside job and then in a competitive job. Despite guaranteed access to set-aside jobs, SEP participants still earned and worked less in all jobs taken together than ACT-IPS participants in a rural labor market with few work opportunities. Our rural work outcomes replicate the findings of prior urban-based trials showing graduated work adjustment as an ineffective strategy for increasing competitive job tenure for most adults with SMI and little job experience.²²

Patterns of Competitive Work Over Time

Fourth, work patterns varied markedly across participants and over time. One-third of participants did not hold a job during the study period. Participants who worked manifested wide variation in job tenure and time taken to obtain their first competitive and any paid job. Such heterogeneous work patterns resist easy explanation. Each participant's initiation, pace, and nature of recovering work capacities may follow a unique and nonlinear trajectory over time, independent of service receipt.⁴² For those who did receive considerable services over extended periods, service impacts on work activity might lag differentially across participants over time. However, uneven work patterns might also be partly a function of participants' employment goals, which ranged from little or no work to full-time work expected by federal social assistance programs and recent “phase-ins” of Ticket-to-Work and Work Incentives Improvement Act (1999, Public Law 106-170) provisions. Because the factors associated with dissimilar work trajectories over time are not well known, we recommend identifying potent personal, service, and environmental contextual factors that may drive and/or inhibit vocational recovery paths over time.⁴³ Findings will help employment specialists to better match the selection and timing of services to each person's unique and evolving recovery status, strengths, and work preferences.

Association of Service and Participant Characteristics With Competitive Work Outcomes

Fifth, controlling for the time-varying effects of total vocational and mental health contacts did not substantively alter the effects of time, program, and their interaction on competitive work probabilities and income earned over time. However, “total service contacts” is a blunt time-varying measure that does not capture service integrative functions, as described in prior SE process analyses, and may therefore lack construct validity.

Two of 7 preselected participant characteristics, diagnosis (schizophrenia) and federal benefit status (currently on the rolls), were associated with the lowest likelihood of working and the lowest income over time, consistent with findings of prior studies.^{23, 44–46} Because no interactions between the 7 characteristics and program on work probabilities and earned income achieved statistical significance, they cannot be inferred, in this study sample and setting, as moderators of program impact.

African American participants, representing 75% of the sample, were half as likely to work compared to whites after accounting for the explanatory effects. Conditional on working, however, both African Americans and whites earned about the same amount of income. Nevertheless, African Americans have historically faced the severest educational, economic, and health disadvantages in the study area. Developing innovative strategies to help them overcome hardships not of their own making, and realize employment goals, must become a policy priority.

Comparison of Competitive Work Outcomes With Published SE Trials

Of the 14 completed randomized clinical trials, individual reports of 8 have been published.^{32–33, 47–52} All 8 trials report percentages of SE and comparison program participants who held at least 1 competitive job during 12- to 24-month study periods. SE program percentages ranged from 27 to 78% compared to 64% in our study. Comparison program percentages ranged from 7 to 40% compared to 37% in our study. Five trials^{32–33, 47–48, 52} report income differences between SE and comparison programs as means (SDs), requiring that we shift from medians (IQRs) to this metric. SE program earnings per month ranged from $37 to $189 compared to $114 in our study. Comparison program earnings per month ranged from $3 to $60 compared to $39 in our study. Thus, the magnitude of the ACT-IPS program's work probabilities and monthly earned income fell between the median and maximum values of the published trials' SE and comparison programs, despite operating in a rural economy with limited job opportunities, high service-sector unemployment, and high poverty among African Americans. In line with the 8 trials, we found no substantive differences between our study programs on psychiatric symptoms and self-reported quality of life. Because our symptom measure manifested floor effects, we cannot accurately estimate program impact on symptoms.

Limitations of Inference

First, project redesign and deviation from a prespecified random assignment process may have compromised study internal validity and program construct validity. Although sensitivity analyses revealed no substantive differences across a wide spectrum of participant and program variables before and after redesign, we cannot rule out that selection biases, program changes, and unobserved variables confounded between-program comparisons. Second, we did not measure some key constructs associated with work outcomes, particularly working alliance⁵³ and neurocognitive status,⁵⁴ both of which may account for some “unexplained” participant-level variance on competitive work outcomes. Third, the competitive work outcomes of the ACT-IPS program, which integrated vocational and mental health services within a self-contained team, may not be directly comparable to those of freestanding SE programs that integrate their vocational services with independent community center mental health services.

Conclusions

A single program that blended 2 evidence-based practices (ACT and IPS) and tightly integrated vocational rehabilitation with mental health services functioned respectably in a rural service mix and economy. The ACT-IPS program aided persons with SMI-related work impairments, limited job experience, and tremendous educational and economic disadvantage to attain competitive work outcomes on a par with the urban SE programs previously described in the literature. Moreover, the program's service-coordinating functions and immediate attention to competitive work may have been key processes that overcame common rural-area obstacles to receipt of appropriate services and to labor market participation. However, the median participant in this and other SE trials worked 5 to 10 full-time-equivalent weeks of work per year, earning between $1,000 and $2,000 at an hourly wage at or near the federal minimum. To be sure, these outcomes reflect meaningful gains in labor market activity over 1- to 2-year intervention periods, but they fall far short of economic independence. We firmly believe that furthering work activity requires public policy adjustments that reduce barriers to accessing higher education,⁵⁵ focus on career-oriented jobs providing opportunities to master highly marketable “technical skills,”⁵⁶ and, for those who leave federal insurance programs to work, rapidly restore their benefits eligibility in the event of setbacks during ongoing recovery efforts.