Improving Work Outcome in Supported Employment for Serious Mental Illness: Results From 2 Independent Studies of Errorless Learning

Abstract

Background

Heterogeneity in work outcomes is common among individuals with serious mental illness (SMI).

Objective

In 2 studies, we sought to examine the efficacy of adding errorless learning, a behavioral training intervention, to evidence-based supported employment to improve SMI work outcomes. Work behavior problems were targeted for intervention. We also explored associations between early work behavior and job tenure.

Methods

For both studies (VA: n = 71; community mental health center: n = 91), randomization occurred at the time of job obtainment with participants randomized (1:1) to either errorless learning plus ongoing supported employment or ongoing supported employment alone and then followed for 12 months. Dependent variables included job tenure, work behavior, and hours worked and wages earned per week. For the primary intent-to-treat analyses, data were combined across studies.

Results

Findings revealed that participants in the errorless learning plus supported employment group stayed on their jobs significantly longer than those in the supported employment alone group (32.8 vs 25.6 wk). In addition, differential treatment effects favoring errorless learning were found on targeted work behavior problems (50.5% vs 27.4% improvement from baseline to follow-up assessment). There were no other differential treatment effects. For the prediction analyses involving work behavior, social skills explained an additional 18.3% of the variance in job tenure beyond levels of cognition, symptom severity, and past work history.

Conclusions

These data support errorless learning as an adjunctive intervention to enhance supported employment outcomes and implicate the relevance of workplace social difficulties as a key impediment to prolonged job tenure.

Introduction

Difficulties obtaining and maintaining employment have been a longstanding problem for individuals with serious mental illness (SMI). Supported employment is a well-established, evidence-based practice designed to promote employment in SMI.¹ Core principles of the Individual Placement and Support model (IPS) of supported employment include rapid job search, integration with mental health treatment services, and continuous follow-along job support.² Results from randomized control trials indicate that job obtainment rates generally fall 2 to 3 times higher for supported employment compared to traditional vocational rehabilitation.³ However, job tenure is more variable with mean job duration ranging from over a year to as little as 10 weeks across studies.^4–8

One approach to augmenting the benefits of supported employment is to identify key determinants of work outcome and target them for treatment. One such determinant, impairments in learning and skill acquisition, may be particularly relevant. A type of rehabilitation intervention not previously examined in studies of IPS supported employment is errorless learning. Errorless learning is a method of training with origins in the experimental psychology and behavioral learning literatures. It is based on the premise that learning is stronger and more durable if it occurs in the absence of errors.^9,10 Across clinical disorders, studies show that the acquisition of new learning in cognitively impaired individuals is impeded by the commission of errors.^11–15 The methods involved in errorless learning are designed to teach new skills through a series of discrete training steps that are graduated in difficulty level and designed to minimize the commission of errors at each step of the to-be-learned task.¹⁶

In 2 independent studies (at the VA and a community mental health center), we tested the effects of adding errorless learning training to ongoing IPS supported employment for the purpose of enhancing work outcomes. We hypothesized that participants who received errorless learning plus IPS supported employment would have longer job tenure and greater improvements in work behavior than participants who received IPS supported employment alone. We also explored the degree to which work behavior within the first 4 weeks after job obtainment predicted subsequent job tenure.

Methods

Participants

The VA study included 74 Veterans who met Diagnostic and Statistical Manual-IV (DSM-IV) criteria for schizophrenia or schizoaffective disorder. The community-based study included 106 participants who met Substance Abuse and Mental Health Service Administration (SAMHSA) criteria for SMI as well as the study’s criterion for memory impairment (ie, greater than 1 SD below a T-score of 50 using age- and gender-corrected norms) on the Hopkins Verbal Learning Test—Revised.¹⁷ Because of the focus on paid work outcomes, participants in either study whose primary interest was school or volunteer work were not included in the analyses (VA: n = 3; community mental health center: n = 15). Diagnoses were based on the Structured Clinical Interview for DSM-IV (SCID-Axis I Disorders/Patient Edition¹⁸) by diagnostic interviewers trained to reliability standards within the MIRECC Treatment Unit.¹⁹ All participants were recent enrollees in their site’s IPS supported employment program (including 6 participants at the VA who were enrolled in an affiliated VA Transitional Work Experience program). Other selection criteria included: (1) age 18 and upward; (2) no history of neurological disease; (3) no history of head trauma with loss of consciousness greater than 1 hour; (4) no alcohol or substance dependence in the past 3 months; and (5) clinical stability (ie, no inpatient hospitalizations in past 6 months, no changes in antipsychotic medication in the past 4 weeks). Medication type and dose were determined by the treating psychiatrist. See table 1 for characterization of each site’s participants who obtained jobs. For both studies, written informed consent was obtained from each participant; research procedures were approved by each study’s respective Institutional Review Board (VA, UCLA).

Table 1.

Demographic Characteristics of Study Participants Who Obtained Jobs

	VA			Community MHC
Demographic/Clinical Variable	EL + SE (n = 11)	SE alone (n = 13)	Group Comparison (P Value)	EL + SE (n = 17)	SE alone (n = 17)	Group Comparison (P Value)
Age (y)	49.6 (11.7)	46.1 (12.4)	ns	40.8 (13.0)	34.1 (9.5)	.10
Education (y)	13.2 (1.4)	12.5 (2.2)	ns	11.8 (2.8)	11.6 (3.2)	ns
Gender (M:F)	10:1	13:0	ns	13:4	13:4	ns
Ethnicity (% white)	9.1	38.5	.10	11.8	5.9	ns
Illness chronicity (y)	17.9 (9.5)	22.6 (15.3)	ns	14.8 (13.1)	14.7 (9.0)	ns
MCCB (overall composite T-score)	28.9 (7.4)	32.7 (13.7)	ns	27.4 (12.3)	31.0 (14.0)	ns
BPRS (total score)	44.9 (8.2)	41.4 (8.1)	ns	40.0 (11.4)	38.9 (10.7)	ns

Note: MCCB, MATRICS Consensus Cognitive Battery; BPRS, Brief Psychiatric Rating Scale; MHC, Mental Health Center; EL, errorless learning; SE, supported employment.

Procedures

For both studies, randomization occurred at the time of getting a job, with participants assigned (1:1) to either errorless learning plus ongoing IPS supported employment or ongoing IPS supported employment alone (see Consort diagram in figure 1). Participants in both groups had specific work behavior problems targeted for intervention which were prominent areas of difficulty noted during the first 3 to 4 weeks on the job as measured by the Work Behavior Inventory (WBI).²⁰ Types of problems included subpar quality of work (eg, floors remained dirty after mopping) or interpersonal difficulties (eg, doesn’t talk to supervisor or co-workers). Given the studies’ targeting of empirically-derived work behavior problems, a study procedure superimposed on usual IPS service practices, these programs are best characterized as “augmented” IPS supported employment. Targeted work behavior problems were addressed in the following ways according to group.

Fig. 1.

Consort diagram.

Errorless learning training followed previously developed manualized procedures which included collection of baseline and post-training data as well as behavioral data during training to monitor error commission.²¹ An individualized training plan was developed for each participant with the targeted behavior broken down into its constituent elements. A series of training steps were then devised, arranged hierarchically by level of difficulty or by their natural order of sequence. Instructional aids (eg, cues, prompts, modeling, self-instruction) were included at each step to ensure high levels of performance proficiency throughout the course of training (see case example in supplementary material). These aids were then withdrawn over time to facilitate functional independence. For the VA study, employment specialists conducted errorless learning training at the job site; for the community-based study, an experienced research staff person conducted training at the clinic.

Supported employment methods to address targeted WBI problems were not prescribed but left to the employment specialists’ discretion. These methods included providing general support, implementation of problem-solving strategies and role-play exercises, and utilization of outside resources, procedures which have been documented as ones commonly used in other IPS programs.²² For the VA study, employment specialists implemented the intervention; for the community-based study, implementation was performed by a research staff person with IPS expertise. Meetings were held with the PI weekly or biweekly to document method type and time devoted to implementation.

Participants who obtained jobs were followed for 12 months from the time of their job start date. The WBI was administered at baseline (3 to 4 weeks after the job start date) and after completion of errorless learning training/IPS intervention by raters who were blind to group assignment. Other work outcome data (hours worked, wages earned) were collected cumulatively over the 12-month follow-up.

Assessment Measures

The WBI measures 5 work skill areas (social skills, cooperativeness, work habits, work quality, and personal presentation) rated on a 1 to 5 Likert scale (1 = consistently an area needing improvement; 3 = performance adequate in this area; 5 = consistently an area of superior performance), and was administered as a semi-structured interview given the context of competitive employment in community-based jobs. For the VA study, it was administered to the Veteran’s work supervisor; for the community-based study, it was administered to study participants.

After starting their new job, participants also received an assessment of cognition using the MATRICS Consensus Cognitive Battery (MCCB)²³ and psychiatric symptom severity using the 24-item Brief Psychiatric Rating Scale (BPRS).²⁴ For the MCCB and BPRS, the primary derived measures were an overall composite and a total score, respectively.

Statistical Analyses

All analyses were conducted using SPSS v.24. The primary analyses included all randomized participants and used an intent-to-treat model to examine the efficacy of adjunctive errorless learning plus IPS supported employment vs IPS supported employment alone at improving job tenure (weeks worked), work behavior (scores from WBI items which were targets of training), and other measures of work outcome (hours worked per week, wages earned per week).

For job tenure, a survival analysis was conducted using the Kaplan-Meier procedure to estimate the overall survival functions (rates across time) for each group and the Log-Rank test to compare the groups. The target event was job loss. Two subjects, one from each group, were censored at the point that they left the study due to extraneous factors that interfered with their employment trajectory (moved out of state, personal injury). Group comparisons of job retention status based on proportions still working at the end of the 12-month follow-up period were analyzed using a Z test. To examine the effects of site and the treatment group by site interaction we conducted follow-up survival analyses using a Cox proportional hazards model, which allows for the entry of covariates. The other work outcome measures (hours worked per week, wages earned per week) were log-transformed due to skewness of the distributions and then analyzed using 2 (group) × 2 (site) ANOVAs.

For examination of training effects on targeted work behavior problems, we conducted a 2 (group) × 2 (time) × 2 (site) repeated measures ANOVA using the SPSS linear mixed model procedure which uses a restricted maximum likelihood procedure to estimate the parameters of the analysis of variance and does not require complete cases. The covariance structure was a random effects platform with compound symmetry: correlation metric. Group and site were between-group variables; time (baseline vs follow-up assessment) was the repeated measure within-group variable. The dependent variable was the mean score for targeted WBI items. Training efficacy would be supported by a significant group × time interaction.

Secondary analyses were conducted to examine putative determinants of supported employment work outcomes. These analyses were conducted to: (1) assess differential areas of strength vs weakness in work behavior across SMI participants during the first few weeks of a new job and (2) assess the strength of the relationship between early work behavior and subsequent work outcomes. For (1), we conducted a repeated measures ANOVA with skill area as the within-subjects factor. Any significant overall effects were followed up with pair-wise contrasts to test for significant differences between each of the 5 work skill areas. Bonferroni correction was employed to control for multiple comparisons. For the weakest area(s) of work behavior, we conducted hierarchical multiple regression analyses to examine the amount of variance in job tenure (weeks worked) explained by work behavior beyond other putative determinants (cognition, psychiatric symptoms, work history). Cognition (MCCB overall composite), psychiatric symptoms (BPRS total score), and work history (highest level of previous employment) were entered in step 1 and the weakest work skill area was entered in step 2. For all of the above analyses (primary and secondary), data were combined across studies/sites. For any significant results, follow-up contrasts were conducted to examine site effects.

Results

General Findings

Fifty-eight participants across the 2 studies got jobs (VA = 24; Community mental health center = 34). The majority of jobs were part-time and paid minimum wage or slightly higher. Examples included janitor, housecleaner, landscaping assistant, clerical assistant, driver, and retail store clerk among others. Of the participants who got jobs, 32 received errorless learning training or IPS intervention (VA = 14; Community mental health center = 18; see figure 1). The primary reason for not receiving training or intervention was early job termination (ie, quit or got fired from job shortly after job start date; n = 15). Other reasons included scheduling difficulties due to full-time job (n = 4), obtained job too close to study termination (n = 3), refused training (n = 1), transportation problems (n = 1), no identified targets (n = 1), and marked conceptual disorganization (n = 1). The mean number of sessions completed was 4.6 (SD = 2.9) for the errorless learning group compared to 2.9 (SD = 1.5) for the IPS supported employment group, which fell at a trend level difference (t(30) = 1.685; P = .10).

Errorless Learning Efficacy

Job Tenure, Hours Worked, Wages Earned.

The results of the log rank test for job tenure based on the Kaplan-Meier survival functions revealed a statistically significant difference favoring the errorless learning plus supported employment group over the supported employment alone group (log rank χ² = 4.039; df = 1; P = .044; see figure 2). Estimated mean job tenure based on the survival analyses which properly account for censoring indicated that participants in the errorless learning plus supported employment group worked 32.8 weeks (SE = 3.8) compared to 25.6 weeks (SE = 3.4) for the supported employment alone group (median weeks for errorless learning plus supported employment = 37; median weeks for supported employment alone = 26). Comparisons for job retention status at the end of the 12-month follow-up (excluding the 2 early censured subjects) revealed that 40.7% of the errorless learning plus supported employment group (11/27) were still continuously working compared to 13.8% of the supported employment alone group (4/29) which was statistically significant (Z score = 2.27, P = .02). The Cox regression analyses revealed no significant site or interaction effects. There were no group differences on hours worked or wages earned.

Fig. 2.

Survival analysis results for errorless learning plus supported employment vs supported employment alone on weeks worked over the 12-month follow-up period.

In sum, these findings indicate that both groups worked approximately the same number of hours per week and made approximately the same amount of money per week, but the errorless learning plus supported employment group was more likely to hold onto their jobs for the full 12-month follow-up period compared to the supported employment alone group.

Targeted Work-Related Behavior Problems.

The results of the mixed model 2 (group) × 2 (time) × 2 (site) repeated measures ANOVA on targeted work problems revealed a significant group × time interaction with the errorless learning plus supported employment group showing greater improvement on targeted work problems from baseline to follow-up assessment than the supported employment alone group (F(1,35) = 4.115; P = .05; see figure 3). There were also significant effects of time with both groups improving on targeted problems from baseline to completion of training (F(1,35) = 44.641; P = .0001) and site with the community mental health center showing more severe levels of work behavior problems overall (F(1,35) = 19.908; P = .0001).

Fig. 3.

Change in work behavior inventory scores for targeted training problems from baseline to follow-up assessment.

Differential Levels of Impairment in Work Behavior Skill Areas

The results from the examination of baseline differences on WBI work skill areas revealed a significant overall effect (F(4,49) = 17.470; P = .0001) with Social Skills and Work Quality being the areas of greatest weakness (all Ps < .014 corrected for multiple comparisons; see supplementary figure S1). This pattern of results was observed at both sites.

Work Behavior as a Predictor of Job Tenure

In the hierarchical multiple regression analyses, we examined the amount of variance explained by Social Skills and Work Quality beyond cognition, overall symptom severity, and highest level of previous work history on weeks worked. The results showed that Social Skills explained an additional 18.3% of the variance beyond the contribution of the other variables in the model (table 2). By contrast, Work Quality explained only 4.4% additional variance. These findings were observed similarly at both sites.

Table 2.

Results of Hierarchical Regression Analyses for Work Behavior Inventory Skill Areas of Social Skills and Work Quality on Weeks Worked

Model 1 With WBI Social Skills
Predictor	β	t-value	P-Value	R 2
Block 1				.001
Cognition	.021	0.111	.912
Overall symptom severity	.025	0.132	.896
Work history	−.001	−0.007	.995
Block 2				.184
Cognition	−.029	−0.161	.874
Overall symptom severity	.232	1.199	.240
Work history	−.117	−0.656	.517
WBI Social Skills	.695	2.800	.018
Model 2 With WBI Work Quality
Predictor	β	t-Value	P-Value	R 2
Block 1				.001
Cognition	.021	0.111	.912
Overall symptom severity	.025	0.132	.896
Work history	−.001	−0.007	.995
Block 2				.045
Cognition	−.026	−0.134	.894
Overall symptom severity	.097	0.487	.630
Work history	−.065	−0.333	.741
WBI Work Quality	.238	1.136	.265

Note: β, standardized beta; WBI, Work Behavior Inventory. Block 2 included the independent variables retained from the previous regression step plus the independent variables added to the model to examine amount of additional variance explained.

Discussion

Although supported employment under the IPS model is a highly regarded and recommended rehabilitation practice for improving employability for individuals with SMI,²⁵ heterogeneity in job tenure persists. Our findings, using combined data from 2 independent studies, provide support for the benefits of adjunctive errorless learning training to augment IPS supported employment. Participants randomized to errorless learning plus ongoing supported employment had longer job tenure and lower job termination rates at a 12-month follow-up compared to ongoing supported employment alone.

In theory, it is believed that the advantages of errorless learning are tied primarily to the engagement of selected areas of implicit memory in forming and strengthening stimulus-response connections involved in new learning.^26,27 In a number of neurological disorders, including schizophrenia, explicit memory is ineffective for learning new skills and abilities and therefore requires the involvement of relatively intact implicit memory processes to fulfill this role. Given that implicit memory does not discriminate between errors and correct responses but does increase the likelihood for a previous response to reoccur, training approaches that allow errors to occur serve to strengthen those responses and the likelihood of their being repeated in the future. Two early studies suggested that the errorless learning advantage over trial-and-error methods was due to the effects of error prevention on residual explicit memory capacities.^28,29 However, the preponderance of evidence from laboratory-based studies that specifically addressed the methodological pitfalls from earlier studies support a primary role for implicit memory in errorless learning.^30–32

A key finding from our studies was that social skills difficulties in the workplace were a prominent problem for supported employment participants. Few studies have examined this problem area within the context of work rehabilitation, but there exists some evidence to support its significance.^33–36 For example, in a study that examined mechanisms of work outcome in veterans participating in a 26-week work rehabilitation program, a path analysis model showed that indirect effects mediated by social cognition and social discomfort explained 18.4% of the variance in work outcome compared to only 4.8% explained by neurocognition alone.³⁴

Interpretation of results warrants consideration of the following primary limitation. A substantial portion of participants across both studies quit their jobs before training/intervention was initiated which limited the power to examine the results from either study alone. We determined that an intent-to-treat model was the most rigorous way to analyze the combined data, even for the WBI data, despite the fact that only a subset of participants had their work-related problems addressed through intervention. It would have been possible to analyze only those who received training/intervention. However, participants were not randomly assigned to training vs non-training (which was chiefly based on length of time involved in the job), so analyzing the treatment effect of only those participants who received errorless learning training or IPS intervention would have introduced a bias. To place early job termination in context, it should be noted that IPS does not aim for continuous employment but rather encourages participants to change jobs when the fit or their satisfaction with the job are not good. Even so, the percentage of participants in these studies who quit their jobs shortly after beginning them was striking and highlights the need to address skill deficits related to frequently occurring work-related problems. Other limitations included methodological differences involving WBI administration and personnel involved in training/problem resolution, as well as the studies’ small sample sizes.

To conclude, although challenges remain, the results from these 2 studies indicate that heterogeneity in job tenure can be reduced with adjunctive interventions such as errorless learning training that augments the benefits of IPS supported employment services. Moving forward, the results on social difficulties as a factor associated with early job termination indicate that this area of functioning should be a consideration for future efforts to improve SMI work outcomes.

Supplementary Material

Supplementary material is available at Schizophrenia Bulletin online.

Funding

This work was supported by the National Institutes of Health (R01 MH082939) and the Department of Veterans Affairs (D4505I).