Implementation of Evidence-Based Employment Services in Specialty Mental Health

Abstract

Objective. Study a quality improvement approach for implementing evidence-based employment services at specialty mental health clinics.

Data Sources/Study Setting. Semistructured interviews with clinicians and administrators before, during, and after implementation. Qualitative field notes, structured baseline and follow-up interviews with patients, semistructured interviews with patients after implementation, and administrative data.

Study Design. Site-level controlled trial at four implementation and four control sites. Hybrid implementation–effectiveness study with mixed methods intervention evaluation design.

Data Collection/Extraction Methods. Site visits, in-person and telephone interviews, patient surveys, patient self-assessment. A total of 801 patients completed baseline surveys and 53 clinicians and other clinical key stakeholders completed longitudinal qualitative interviews.

Principal Findings. At baseline, sites varied in the availability, utilization, and quality of supported employment. Each site needed quality improvement for this service, though for differing reasons, with some needing development of the service itself and others needing increased service capacity. Improvements in knowledge, attitudes, beliefs, and referral behaviors were evident in mid- and postimplementation interviews, though some barriers persisted. Half of patients expressed an interest in working at baseline. Patients at implementation sites were 2.3 times more likely to receive employment services during the study year. Those who had a service visit were more likely to be employed at follow-up than those who did not.

Conclusions. Studies of implementation and effectiveness require mixed methods to both enhance implementation in real time and provide context for interpretation of complex results. In this study, a quality improvement approach resulted in superior patient-level outcomes and improved clinician knowledge, attitudes, and behaviors, in the context of substantial variation among sites.

Schizophrenia occurs in 1 percent of the population and results in substantial morbidity and mortality when evidence-based treatments are not provided (Lehman 1999; Weeks, Yano, and Rubenstein 2002; Perala et al. 2007). Supported employment (SE) is one of eight evidence-based treatments recommended in the Schizophrenia Patient Outcomes Research Team treatment guidelines (Dixon et al. 2010). Any person with schizophrenia who expresses a desire to work is eligible for and should be offered SE with the goal of competitive employment (Dixon et al. 2010). Unfortunately, the impact of SE has been muted due to variable fidelity to the model and limited service utilization.

The Veterans Health Administration of the Department of Veterans Affairs (VA) implemented SE starting in late 2003. The VA is divided into 21 regions nationally, called Veterans Integrated Service Networks (VISNs). The VA established a SE mentor-trainer site in each VISN. These mentor-trainer sites used a train-the-trainer model to establish further sites. To this day, sites are periodically monitored through site visits and ratings of fidelity to the SE model; but prior to the current study, no quality improvement (QI) efforts targeted utilization of SE.

It is well-established that receipt of SE is the single best predictor of employment among patients with serious mental illness (Bond, Drake, and Becker 2008), but barriers to its utilization persist (Bond et al. 2001). In the United States, fewer than 5 percent of patients with schizophrenia who could benefit from SE have access to it (Wang, Demler, and Kessler 2002; West et al. 2005; Bond and Drake 2008). Within the VA, there are organizational obstacles that could hamper access and utilization. These include long-standing programs in work therapy, incentive therapy, and sheltered workshops, which are in conflict with the principles of SE (Drake, Bond, and Essock 2009); clinicians naive to the eligibility requirements of SE; limited time in patient–clinician visits for discussion of employment interest and SE; and a lack of information regarding how many patients want SE. Other obstacles stem from the model requirement of a small caseload and time-unlimited support, which necessitates costly resources. Finally, lukewarm attitudes toward evidence-based services such as SE are known to influence implementation (Marshall et al. 2008); Lehman (2010) has referred to this as a “hesitation waltz” around the adoption of these services.

Several studies have identified barriers and facilitators to SE implementation, but few have used mixed methods to characterize implementation in mental health care, and none have actively targeted them as part of a QI effort. “Enhancing QUality In Psychosis” (EQUIP) was a clinic-level controlled trial which sought to increase appropriate utilization of SE for patients with schizophrenia. EQUIP utilized a “hybrid type 2” effectiveness-implementation study design, which balances attention to the effectiveness of the clinical intervention and implementation strategy to support the intervention (Brown et al. 2008; Curran et al. 2012). Guided by the Simpson Transfer Model of organizational change (Lehman, Greener, and Simpson 2002), EQUIP evaluated effectiveness and implementation using mixed methods during four stages: exposure, adoption, implementation, and practice. Qualitative data guided use of implementation strategies by proactively responding to preimplementation and midimplementation findings related to clinicians' and administrators' knowledge, attitudes, beliefs, and behaviors related to SE services. Qualitative results from all time points were used to contextualize the outcomes evaluation. The combination of datasets allowed for a more comprehensive understanding of the utilization and impact of SE.

Methods

Study Design

This clinic-level controlled trial was conducted in four VISNs. Leadership in each VISN named a pair of specialty mental health clinics that were matched on academic affiliation (known to affect organizational engagement in QI; see Weeks, Yano, and Rubenstein 2002, Yano 2000) and number of patients with schizophrenia. One was assigned to implementation and one to control (usual care), for a total of four implementation and four control sites. A multifaceted implementation strategy (Powell et al. 2012) included activities at the patient, provider, and organizational levels (Table 2; Cohen et al. 2013).

The effectiveness evaluation began in January 2008 when clinicians and patients began enrollment and completed a baseline survey. Patient enrollment lasted an average of 13 months. Final patient surveys began in May 2009. The implementation evaluation occurred in three waves, the timing of which was site-specific based on dates of start-up and patient enrollment. Pre-implementation interviews began in late 2007; midimplementation, mid-2008; and postimplementation, mid-2009. This design has been described as a basic convergent design within an intervention mixed methods framework (Fetters, Curry, and Creswell 2013, this issue).

Participants

Patients were eligible to participate if they (1) were at least 18 years old; (2) had a diagnosis of schizophrenia or schizoaffective disorder; and (3) had at least two mental health clinic visits during a 6-month eligibility period. From the overall population of eligible patients, a random sample was identified at each site. Probability of inclusion was based on the overall eligible population, desired sample size, and expected nonparticipation. Eligible veterans were approached in person at clinic visits. A total of 1,964 patients were eligible, 530 were not approached, 633 refused to participate, and 801 consented to be enrolled (41 percent). A random sample of implementation site patients (83/295; 28 percent) also completed a 10–15 minute qualitative interview postimplementation.

Clinicians were eligible to participate if they treated eligible patients. Mental health administrators of the enrolled clinics were also eligible. Two hundred and one clinicians and administrators consented to participate in the study and complete the organizational survey (Hamilton, Cohen, and Young 2010). Key stakeholders at implementation sites also completed a qualitative interview at baseline (preimplementation), and, when possible, at mid- and/or postimplementation. At baseline, 16 administrators and 43 staff completed the survey; of those, 38 (64 percent) also completed a qualitative interview. At midimplementation, 22 completed the interview and at postimplementation, 35. Twenty-seven individuals completed at least two of the three interviews. The sample fluctuated over time due to availability of respondents as well as turnover in various roles. At postimplementation, the sample expanded to include more employment specialists.

Measures

Mixed methods were used to evaluate implementation and effectiveness, relative to usual care (see Table 1). Semistructured interview guides were used for all three waves of qualitative data collection. The preimplementation guide focused on knowledge of existing structures and practices related to SE (e.g., staffing, referral processes) and attitudes and beliefs regarding competitive employment among patients with SMI. The midimplementation guide inquired as to whether respondents observed changes in the clinic attributable to EQUIP, as well as changes in SE structures and practices. The postimplementation guide queried perceptions of the overall impact of the project. The postimplementation interview of employment specialists focused on their training and job. The postimplementation interview of patients inquired, in part, about their experience with SE.

Table 1.

Mixed Methods Data Collection

Data Types	Data Source	Sample Content
Semistructured interviews	Clinicians, administrators, patients	Participation, level of implementation, satisfaction
Field notes	VISN coordinators	Group-level dynamics, implementation details
Patient kiosk self-assessments and research assessments	Patients	Demographics, service need and utilization, psychiatric symptoms
Administrative data	Electronic medical record	Visit dates, treatments
Organizational readiness surveys	Administrators and staff	Organizational climate, readiness for change, burnout
Activity logs	Quality coordinators (RNs)	Time spent by staff on clinical interventions

Shaded cells: data included in present analyses. VISN, Veterans Integrated Service Networks; RN, registered nurse.

In terms of quantitative data, at baseline, patient diagnosis was confirmed using an abbreviated version of the Structured Clinical Interview for the DSM-IV (First et al. 1995). Current symptoms were rated using the Brief Psychiatric Rating Scale (Ventura et al. 1993). Research assistants (RAs) administered the baseline interview after being trained to a high level of reliability. Routine quality checks were completed (Ventura et al. 1993). It was not possible to blind interviewers to clinic assignment. To reduce bias, interviewers had minimal or no contact with staff involved with study implementation. Structured chart reviews were completed for each patient using the electronic medical record. Visits that included either the development of a vocational assessment profile, a strategic employment plan, or ongoing job development (Bond and Drake 2008) were counted. The total number of SE visits for the year prebaseline and the year postbaseline was computed for each patient.

Data Collection

Clinicians and administrators at implementation sites (n = 53, cumulative) were interviewed either in-person or by phone by the lead author, an experienced qualitative researcher, or by the VISN study coordinators (all PhDs) who were trained in interviewing by the lead author. The vast majority of the interviews were recorded and professional transcribed.

Patients completed baseline (n = 801) and follow-up research assessments (n = 662; 83 percent). Patients at implementation sites also completed a self-assessment at a patient-facing kiosk each time they had a mental health clinic visit (Cohen et al. 2013). Kiosks assessed clinical status, treatment preferences, and receipt of treatments, and made this data available to patients, providers, and for QI. At postimplementation, RAs conducted semistructured interviews with a random sample of implementation site patients (n = 83). They were instructed to document patients' words verbatim when possible; transcription was not available for these interviews.

Written informed consent was obtained from all enrolled patients and staff. The study was approved by the Institutional Review Boards of all participating sites.

Data Analysis

Interview data were analyzed iteratively. After pre- and midimplementation interviews, transcripts and field notes were analyzed primarily by the lead author, using a hybrid deductive/inductive thematic analysis approach (Fereday and Muir-Cochrane 2006), using codes related to key topics as well as codes that emerged from the data (Ritchie and Spencer 1993). In general, inductive codes were applied when they pertained to at least 50 percent of the data. For example, a deductive primary code was SE; secondary codes included SE referral processes, SE capacity, SE utilization, SE benefits, and familiarity with SE. Inductive codes related to employment mainly pertained to beliefs about patients' ability and desire to work. Text segments coded by the lead author were reviewed by the second author; disagreements, which were rare, were resolved through clarification and discussion. Axial coding (Strauss and Corbin 1990) was performed to examine relationships among employment-related codes and other codes, for example, mental health recovery orientation of the clinic.

Each wave of data was analyzed on its own, and then for the purposes of this and other manuscripts, codes from each wave were compared with one another to examine change over time. All analyses of clinician and administrator interview data were facilitated by ATLAS.ti qualitative data analysis software (Scientific Software Development 2009), which provides multiple approaches to analyzing longitudinal and mixed qualitative data sources (e.g., interview data and field notes). Patient interview data (short answers) were entered into an Excel spreadsheet to tabulate how many patients described experiences with SE and to obtain brief descriptions of those experiences.

All quantitative analyses were conducted using Statistical Software Package SAS Version 9.1.3 (SAS Institute Inc. 2008). Analyses included only those who met the criteria for referral to SE services (interest in competitive employment). Baseline characteristics between implementation and control groups were compared using t-tests for continuous and chi-square analysis for categorical variables.

Frequencies were examined to understand the flow of individuals from expressed interest in employment to SE utilization, and then employment. A logistic regression was conducted using intervention status (control, implementation) to predict presence of an SE visit (yes = 1, no = 0). Demographics (age, African American race, male gender, college graduate or higher degree) were entered as covariates in the model. Implementation and control sites were paired geographically; therefore, a second logistic regression model was conducted to examine differences between and within pairs of sites. Group, pair, and interaction of group by pair were entered as predictors for this model. Age (the only significant demographic predictor) was also used as a covariate. A chi-square analysis was used to compare final employment status between those who utilized employment services and those who did not.

Results

Preimplementation Evaluation

Interviews revealed that the availability, utilization, and quality of SE varied across sites. There was consensus that each site was in need of QI for SE. Site A respondents were familiar with traditional employment services, but not typically with SE. Even in cases where the interviewer specifically asked about SE, respondents were not familiar. One said, “Yeah, it's compensated work therapy…that's the work program. There's incentive therapy and compensated work therapy, and that's it.” Another knew very little about SE, and said, “It's certainly not big here.” Respondents recognized that traditional services were insufficient because they were not “geared to really severe, persistent mental illness,” and expressed that they could “do better” in this area. Respondents perceived that there was an “untapped” population of patients interested in working, but some respondents expressed that patients with schizophrenia had too much difficulty getting or keeping jobs.

Site B respondents were also largely unfamiliar with SE, and most felt that there were many patients who wanted to work but were not working. Those few who were familiar with SE expressed that barriers included patients' concern about disability benefits and clinicians' lack of knowledge. More education was suggested. One respondent noted, “I do think we have some really good programs in place but I don't think the word is really out. So many of the Vets really do feel like that if they start to work, they're gonna lose all their benefits.”

At Site C, all respondents mentioned SE as an available option. When asked what they do when patients with schizophrenia express an interest in working, some respondents immediately mentioned SE, while others mentioned taking an incremental approach, starting with further assessment, and sometimes incentive therapy. Respondents uniformly felt that more could be done to improve utilization. Noted barriers included patient resistance (“some of the patients doubt whether they can work, even if they want to”) and concerns about benefits, as well as clinicians' beliefs that their patients are not ready to work (what one called the “traditional way of thinking”). One respondent noted, “What I think tends to happen is the providers make a decision about the readiness of an individual and then that influences whether they even present the information.” When asked what he would do to get more patients back to work, one respondent simply answered, “Give patients the option.”

At Site D, respondents described a “culture change” associated with the arrival of the employment specialist a year prior: “I think that there's more open-mindedness on our part to getting people into work. [The specialist was informed] of his [caseload] numbers when he first came in—I think 20–25—and he quickly filled those spots…so staff are using it.” One said that with SE, they were “doing the opposite of the medical model,” and that staff saw the effectiveness of SE with their patients. Across the board, though, respondents were concerned about how the SE specialist would handle a potential increase in referrals due to EQUIP. Some felt optimistic that the increase would motivate hiring additional support: “Hopefully this [project] is gonna prove that there is a need to hire more supported employment specialists who can do this kind of work.” But respondents reported: “I really don't know how many people wanna work or not. I don't even have any real sense.”

Midimplementation Evaluation

Anticipated concerns expressed at preimplementation were realized and identified by clinicians. Data from these interviews were used to make decisions about which implementation strategies to deploy. Some sites were also beginning to see QI.

Site A respondents were generally disappointed or frustrated with the lack of SE services available. One clinician stated that there was a “huge need” to get patients back to work, but no resources to facilitate this. However, efforts were underway to rectify the lack of services. One respondent explained, “I think we're realizing that more of our patients would like to get back to work. So I think that's a real positive. And I think if we get more resources, we will see the benefits, and there will be sort of a change in the mindset of a lot of the clinicians.” Consistent with this perception, one respondent reflected that the lack of SE services may have had something to do with “old-school thinking,” that is, perceptions that patients with schizophrenia should or could not be competitively employed.

Site B respondents discussed a recent in-service conducted by an SE coach. (As an implementation strategy, presentations were developed centrally, distributed to sites, and used for in-service presentations; see Table 2.) This in-service increased awareness among clinicians and was perceived to have resulted in increased SE referrals. One respondent said that prior to EQUIP he “wasn't aware of anybody, anywhere, doing anything like this, to get schizophrenics into a workforce environment.” An SE coach confirmed: “I get more referrals and I've talked to a lot more people, which is helping the Veterans, making them think about going to work. Before, they would not even think about it.” Respondents felt that patients needed more education and empowerment, and as a result more educational in-services were promoted. The most consistent suggestion was to increase the number of employment specialists.

Table 2.

EQUIP-2 Multifaceted Implementation Strategy

Patient level
Routine self-assessment data collection via kiosks	Patients completed self-reports of symptoms, side effects, interest in work; entered weight
Education on care targets	Distributed patient “Fast Facts” sheets via kiosk regarding care targets; clinicians educated patients regarding work and weight
Provider level
Feedback of patient self-assessment data (“kiosk printout”)	Clinicians received patient self-report data at the time of clinic visits
Education	PI and co-PI conducted in-person and virtual presentations; clinician “Fast Facts” sheets distributed regarding care targets; pushed treatment recommendations to clinicians through kiosk printouts
Social marketing	PI and co-PI conducted in-person and virtual presentations; research team sent site-specific “e-quip” emails with brief facts about care targets, updates on local resources and activities; posted flyers in clinics about care targets
External facilitation	Conducted monthly meetings with site coordinators and PIs to address implementation issues; held regular meetings with EBQI team leads (see below)
Organizational level
Project kickoff	PI and co-PI visited each site to launch project; generated enthusiasm and fostered collaboration
Clinical champions	Distributed quality reports and discussed performance with providers; fostered positive attitudes toward care targets in day-to-day interactions and clinic meetings; engaged in implementation troubleshooting on behalf of research team
Multidisciplinary evidence-based quality improvement (EBQI) teams	Local Recovery Coordinators trained in EBQI and then facilitated regular meetings of EBQI teams to address local improvement issues using plan-do-study-act cycles

At Site C, respondents perceived an increase in the number of patients with schizophrenia getting SE. Site leadership postulated that clinicians may not necessarily attribute these changes to EQUIP but noted that the social marketing and consistent discussions of SE (two of our implementation strategies) were influencing clinician behaviors. As one lead stated, “[Staff] may not tie all of it together, but something's [changing].” When asked if EQUIP had helped in getting patients back to work, one individual responded, “I think so because of the individual attention that the patient is getting, really asking them what their interests are [referring to the patient self-assessment].” Another said: “I think this study really started at the perfect time, because with supported employment trying to spread the culture of work in individuals with schizophrenia, this study shows that there's some science behind that whole program, and this just really was like the perfect cousin…to help change the culture as well.” Some expressed that colocating SE in mental health would improve utilization, and as a result it was recommended that employment specialists regularly attend clinical meetings. Respondents also felt that SE capacity could be increased, but that increased capacity would not necessarily result in more jobs, which were scarce due to the depressed economy.

Site D respondents expressed concern about SE capacity. As anticipated in preimplementation interviews, referrals increased as a result of the patient self-assessments, but there was not enough staff to meet the demand: “You have a [patient self-assessment] that generates referrals for supported employment, but you cannot provide the service if you don't have the staff.” At mid-implementation, clinic leadership reported that they had used the patient self-assessment data (an implementation strategy) as evidence that additional specialists were needed. Clinicians from this site asked for more educational tools. There was a specific suggestion to use “patient testimonials in a peer-to-peer approach.” Patient testimonials were then added to educational emails (the “e-quip” emails see Table 2) distributed to all clinicians quarterly throughout implementation.

PostImplementation Evaluation

Variability in terms of perceptions of SE and the impact of EQUIP was evident at postimplementation. At Site A there was clear consensus that SE had not been provided to patients either prior to or during the study. Although interested patients were identified, the service was never made available. One clinician stated, “Supported employment has really not been good here,” and another stated, “It did not go well here.” During the study, there were meetings with mental health and rehabilitation service leaders, but they expressed that there was no funding to hire trained SE staff. Therefore, lacking resources and a clinical champion, the site never supported a move from traditional vocational rehabilitation models to SE.

Site B faced capacity problems throughout implementation. Referrals were perceived to have increased as a result of EQUIP, but insufficient staffing made it difficult to accommodate more patients or consider engaging in additional marketing about the service. Some noted, however, that the project improved communication between mental health and employment services.

At Site C there was consistent agreement that EQUIP had positively impacted the number of referrals to SE. Respondents noted that many patients had never been asked if they were interested in working prior to EQUIP, and the self-assessment kiosks asked that of every patient at every visit, which revealed more interest than expected. When asked if EQUIP had changed her practices, one clinician responded: “Sure, yes, I think about supported employment all the time now for my patients.”

At Site D, respondents reflected on a rise in staff awareness of the importance of work, but these changes were not solely or consistently attributed to EQUIP. Similarly, there were mixed perceptions of whether EQUIP was responsible for the hiring of an additional employment specialist; some felt that the project provided the evidence to support this hire, whereas others felt it had nothing to do with EQUIP. Some respondents did note clinical “discoveries” such as patients who were perceived as unable to work, who then attended SE and ultimately found employment and were working. Some individuals felt that the project was responsible for an increased number of referrals to SE, but this was not necessarily interpreted positively since the employment specialists' caseloads were at capacity and new referrals were waitlisted.

A few patients at implementation sites (n = 11; 13 percent) who completed the brief follow-up interview had experiences with SE. Seven found the services to be helpful; others expressed frustration that jobs were not available. Those who answered why they did not use SE (n = 72) gave reasons such as they “didn't want to work” (n = 20), were “too sick to work” (n = 15), “can't work” (n = 11), or “didn't know assistance was available” (n = 11). The primary reasons were evenly distributed across the sites, with the exception of lack of knowledge of the services, which was concentrated at Site A, where SE was not made available.

Effectiveness Evaluation

Sample

Characteristics of the 406 participants who expressed interest in SE during the baseline survey are shown in Table 3. The average participant was 53 years old, male, either white or African American, not currently married, and had completed high school or some college. There were no significant differences in these baseline characteristics between participants at implementation versus control clinics.

Table 3.

Demographic Characteristics and Service Utilization of Patients

	Sample	Implementation	Control	Test Statistic t Score or χ2
N	406	194	212
Demographics
Age, in years (SD)	53 (9.1)	53 (8.8)	54 (9.4)	0.64 (p = .52)
Male (%)	374 (92.1)	180 (92.8)	194 (92.0)	0.23 (p = .63)
Race
Caucasian (%)	175 (43.1)	90 (46.4)	85 (40.1)	4.67 (p = .10)
African American (%)	187 (46.1)	79 (40.7)	108 (50.9)
Other (%)	44 (10.8)	25 (12.9)	19 (9.0)
Marital status
Presently married (%)	89 (21.9)	37 (19.1)	52 (24.5)	1.76 (p = .18)
Has children (%)	212 (52.2)	94 (48.5)	118 (55.7)	2.11 (p = .15)
Education
Less than high school (%)	43 (10.6)	27 (13.9)	16 (7.5)	4.35 (p = .11)
High school graduate or some college (%)	284 (70.0)	131 (67.5)	153 (72.2)
College graduate or higher degree (%)	79 (19.5)	36 (19.0)	43 (20.3)
Clinical characteristics
Psychosis symptom severity, BPRS (SD)	3.0 (1.9)	2.9 (1.8)	3.2 (1.9)	1.56 (p = .12)
Service utilization
Year prior to baseline
At least one SE visit
N (%)	23 (6.0)	13 (6.8)	10 (4.7)	0.77 (p = .38)
Mean (SD)	2.3 (3.1)	1.7 (0.8)	4.2 (4.3)	1.83 (p = .10)
Study year
At least one SE visit
N (%)	49 (12.0)	32 (16.5)	17 (8.0)	6.98 (p < .01)
Mean (SD)	2.3 (2.7)	1.7 (0.2)	3.5 (1.0)	1.71 (p = .10)

BPRS, brief psychiatric rating scale.

Desire to Return to Competitive Employment

A total of 801 individuals completed a baseline interview. Of those, 406 (51 percent) were interested in returning to competitive employment (Table 4).

Table 4.

Interest in Work, Utilization of Services, and Work Outcomes

			Baseline Interest in Returning to Competitive Employment (n = 801)*			SE Visits during Implementation for Those Interested (n = 406)†		Employment Status at 12-Month FU for Those Interested (n = 406)‡			Employment Status at 12-Month FU for Those Interested and Had an SE Appointment (n = 49)§
Site	Pair	Group¶	Interested (%)	Not Interested (%)	Already Working (%)	Initial SE Appointment, yes (%)	Average No. (SD) of SE Appointments	Competitively Employed (%)	Not Competitively Employed (%)	Lost to FU (%)	Competitively Employed (%)	Not Competitively Employed (%)	Lost to FU (%)
A	1	I	64 (55)	36 (31)	16 (14)	3 (5)	3 (3.2)	3 (5)	47 (73)	13 (20)	1 (33)	2 (67)	0 (0)
E		C	76 (50)	48 (31)	29 (19)	6 (8)	7 (5.5)	3 (4)	64 (84)	8 (11)	0 (0)	6 (100)	0 (0)
B	2	I	40 (48)	39 (47)	4 (5)	15 (38)	1 (0.4)	2 (5)	30 (75)	8 (20)	0 (0)	12 (80)	3 (20)
F		C	30 (50)	12 (20)	18 (30)	1 (3)	1 (0.0)	2 (7)	23 (77)	3 (10)	1 (100)	0 (0)	0 (0)
C	3	I	44 (43)	45 (44)	13 (13)	8 (18)	1 (0.5)	5 (11)	31 (70)	8 (18)	2 (25)	5 (63)	1 (13)
G		C	73 (61)	36 (30)	11 (9)	6 (8)	1 (0.4)	3 (4)	67 (92)	3 (4)	1 (17)	5 (83)	0 (0)
D	4	I	46 (53)	26 (30)	15 (17)	6 (13)	3 (0.4)	2 (4)	33 (72)	11 (24)	2 (33)	3 (50)	1 (17)
H		C	33 (43)	27 (36)	16 (21)	4 (12)	2 (1.0)	4 (12)	21 (64)	8 (24)	1 (25)	2 (50)	1 (25)
Total			406 (51)	269 (34)	122 (15)	49 (12)	2 (2.7)	24 (6)	316 (78)	62 (15)	8 (16)	35 (71)	6 (12)

Note.

^*n = 4 refused to answer this question.

^†n = 3 did not have data on appointments.

^‡n = 4 refused to answer this question.

^§n = 6 did not have data on appointments or refused to answer this question.

^¶I, Implementation site; C, Control site (usual care).

FU, follow-up; SE, supported employment.

SE Utilization

For the 406 interested in returning to work, 23 (6 percent) had an SE visit in the year prior to baseline, and this rate was comparable at implementation and control sites (p = .38) (Table 3). In comparison, 49 (12 percent) had a SE visit in the year of the study; this rate was significantly higher at implementation (16.5 percent) versus control sites (8.0 percent) (p < .01) (Table 3). In the year prior to baseline and in the study year, for those who had a SE visit, the average number of visits was two, and this was comparable at implementation and control sites (p = .10) (Table 3). When predicting a SE visit during the study year, intervention status (control vs implementation) was a significant predictor (χ² = 7.14, p < .01) after controlling for demographics (Table 5). The odds of having an SE visit during the study year was 2.3 times more likely if the individual was at an implementation versus control site. When examining the differences between implementation and control sites within pairs, one set of pairs was significantly different from one another. Pair 2, Sites B and F, were significantly different in terms of likelihood to have at least one SE visit during the study year (χ² = 6.75, p < .01). The odds of having an SE visit during the study year was 20.8 times more likely if the individual was at implementation site B versus control site F.

Table 5.

Variables Affecting Receipt of a Supported Employment Visit during Implementation

Variable	Presence of a Supported Employment Visit
	B (SE)	χ2	p
Intercept	−3.63 (1.05)	11.93	<.01
Demographics
Age	−0.02 (0.02)	2.09	.15
African American race	0.01 (0.24)	<0.01	.95
Male gender	−0.25 (0.25)	1.04	.31
High school or some college	0.03 (0.23)	0.01	.91
Intervention status (control vs implementation)	−0.42(0.16)	7.14	<.01

SE, standard error.

Competitive Employment

Of the 406 interested in returning to work, 6 percent were employed at the final follow-up (Table 4). Of those 406 interested in returning to work who utilized SE during the study year, 16 percent were employed at follow-up. The employment rate difference between those who did and did not attend SE (16 percent vs 6 percent) was significant (p < .001).

Discussion

EQUIP is the largest QI study to date conducted in specialty mental health, with some of the health care system's most challenging patients. Within a representative sample of 801 patients with schizophrenia, half were interested in working. To the best of our knowledge, this is the largest sample from which data regarding patient desire to work have been systematically obtained. Our finding is similar to that of Mueser and colleagues, who found that among 528 patients with schizophrenia who were not working, 61 percent reported interest in working (Mueser et al. 2002). This rate of patient interest is in sharp contrast with national estimates that fewer than 5 percent of patients with schizophrenia who are interested in SE services actually have access to it (Kessler et al. 2003; West et al. 2005). This study was designed to improve identification of patients who are eligible for SE (through routine patient self-assessment) and improve clinicians' awareness of the benefits of work, thereby increasing referrals to and utilization of SE. Our convergent mixed methods design with implementation (qualitative) and effectiveness (quantitative) data substantiate that the project was indeed successful in modest achievement of these goals in the implementation sites, when compared with control sites.

Using a QI framework, a theoretically grounded implementation approach (the Simpson Transfer Model), and a mixed methods intervention evaluation design that provided data iteratively throughout the study, the number of individuals who had a visit to SE more than doubled from the prestudy year to the poststudy year. Individuals at implementation sites had a 2.3 greater chance of having an appointment in the study year compared with those at control sites. Despite these improvements, the percent of individuals seen by SE specialists only reached 12 percent of those who had a desire to return to work. We learned from our implementation evaluation that this gap between expressed interest and receipt of services was related to several complex and often interrelated factors. One of the key barriers to SE utilization was capacity. The SE model requires caseloads of no more than 25 individuals. One site did add an employment specialist during the study, but it was late in the study year and its impact on employment was likely left unrealized by the time of follow-up. In addition to capacity issues, nonrecovery-oriented attitudes and beliefs likely contributed to low receipt of SE services. For example, providers expressed that many of their patients were not able or “ready” to work and others admitted that they did not know how many wanted to work. This is consistent with other studies of SE implementation, which have found that clinicians do not consistently refer to SE even when their patients want these services (Casper and Carloni 2007), and clinicians underestimate the desire for employment-related services among patients with serious mental illness (Crane-Ross, Roth, and Lauber 2000).

In a qualitative study of barriers to SE implementation, Pogoda and colleagues found that the primary barriers in six VA medical centers were clustered around “paternalistic-uninformed” concerns about the ability of persons with serious mental illness to be gainfully employed, and a lack of organizational structures and leadership to promote and integrate the SE program (Pogoda et al. 2011). They suggest that during implementation, organizations would benefit from leadership buy-in and from promotion of these programs and education of mental health teams by knowledgeable and influential stakeholders. Bond and Drake (2008) suggest that barriers to implementation include finance, organization, integration, training, and supervision. In the current study, leadership buy-in was critical to implementation, and it was clear that leadership buy-in without accompanying resources can stymie efforts to increase utilization of SE and, most likely, other evidence-based services. The evidence showed that where SE was strong, attitudes toward employment were generally positive, likely because clinicians were observing the benefits of SE in their patients; where SE was weak or nonexistent, attitudes were generally neutral or pessimistic. It is therefore plausible that the development of strong, well-resourced services has a positive influence on recalcitrant attitudes and beliefs.

Rate of competitive employment is the hallmark of successful SE, and it is well established that sites with high SE fidelity have higher rates of employment (Drake, Bond, and Essock 2009). While our study was not designed to address or improve fidelity, we did examine site fidelity ratings to see if they could help to explain the 16 percent rate of competitive employment at 12-month follow-up. We found that three of the implementation and one of the control sites had fair or good fidelity, and the rest of the sites were not delivering services with fidelity. Despite variable fidelity and only a limited number of SE visits over the course of a year, utilization of SE proved to make a significant difference in the percent employed when compared with those who did not attend but were interested in working. An employment rate of 16 percent is slightly lower than other studies where fidelity was tightly controlled, but it is still very respectable for this population. Future studies could examine and potentially alter other factors related to higher employment rates, such as caseload size and supervisory ratings (Taylor and Bond 2012), and increased service intensity (McGuire et al. 2011).

A limitation of this study is that it was confined to four intervention and four control sites. Also, although sites were drawn from large geographic regions, the pool of available sites was limited, and sites could only be matched on a small number of characteristics. Although this study was designed to maximize the generalizability of results, this cannot be fully assured with this sample size. Resources for research are necessarily limited, and intensive data collection is required for studies that include detailed evaluation of both implementation and effectiveness. Although larger hybrid research projects may rarely be possible, researchers may wish to move to evaluation of broader implementation driven by policy changes, with less intensive data collection.

We conclude by stressing the importance of mixed methods and a multilevel evaluation that taps into multiple vantage points (Palinkas et al. 2011). Within each site, there were varying knowledge of SE, varying beliefs about competitive employment among patients with schizophrenia, and varying attitudes about what impact the project would and did have on this service. Had we limited our data collection to one strata (e.g., leaders), one time point (e.g., post-implementation), or one type (e.g., quantitative), we would have potentially misunderstood many dynamics and misfired with our implementation strategy at each site. Furthermore, had we relied on qualitative data only, we would not have been able to demonstrate that our QI approach positively impacted receipt of evidence-based care among patients with schizophrenia. By presenting an integrated set of methods and findings (O'Cathain, Murphy, and Nicholl 2008), we hope to have contributed to the growing emphasis in implementation science (Yano et al. 2012) on elucidating the intricacies of multilevel implementation studies.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.