Is There a Relationship Between Self-Assessed Fidelity and Self-Reported Employment in the Individual Placement and Support Model of Supported Employment?

Abstract

Objective:

A growing literature across a number of practices, including the individual placement and support (IPS) approach to supported employment, demonstrates that high fidelity implementation increases the chances of achieving desired outcomes. This study examined the relationship between self-reported IPS fidelity and employment outcomes and whether this relationship is maintained over time.

Methods:

78 outpatient programs in New York State provided self-reported fidelity and employment outcome data. Pearson correlations were used to determine the relationship between fidelity scores and competitive employment rates. A mixed-effects model examined the relationship between repeated fidelity and employment measures over time.

Results:

A significant positive relationship between IPS self-reported fidelity and employment was found. The relationship between fidelity and employment was sustained through time (up to one year).

Conclusions:

IPS fidelity self-assessed by program sites is shown to be associated with higher employment rates, which are sustained over time.

Introduction

A focus on the implementation of evidence-based practices has brought with it a focus on the quality – that is fidelity- of implementation. Fidelity scales have been developed to guide implementation efforts to ensure that practices are made available as designed and intended. Implicit in this process is the assumption that high fidelity will yield intended outcomes. But is this assumption accurate?

Across many fields and practices, studies have shown a relationship between fidelity and outcome. For example, positive relationships have been found between outcomes and fidelity with assertive community treatment in dual disorders programs (1), 12 step facilitation programs (2), the Housing First model (3), antibullying programs in elementary schools (4), family support programs for improving educational outcomes for at risk youth (5), and middle school obesity programs (6).

The individual placement and support (IPS) model of supported employment (7,8) is an evidence-based practice whose implementation is guided by a clearly designed fidelity scale (9). In two recent studies, Bond and colleagues found that total fidelity score was significantly associated with the quarterly competitive employment rate. The first study (10) examined 79 sites in eight states that participated in an IPS Learning Community administered by the IPS development group and reported a correlation r= 0.34 (p< .01). The second study (11), which aimed to replicate the analyses in (10), using a different set of 79 sites in twelve states in the IPS Learning Community, reported r = 0.27 (p = 0.02). Additionally, a recent study (12) of 21 supported education and employment programs sponsored by the Massachusetts Department of Mental Health (DMH) that were developed in conjunction with the IPS developers to align with IPS principles, found that clients in high IPS fidelity programs (scores ≥65 on the 15 item earlier version of the IPS fidelity scale) were more likely to obtain jobs and to work 20 or more hours per week.

This study examines the relationship between IPS fidelity and employment outcomes, in programs implemented using the New York State (NYS) learning collaboratives, over time, using self-assessed fidelity ratings.

Methods

Our work facilitating the implementation of IPS in programs across NYS using learning collaboratives has been described previously (13). For this study, 78 of the 98 community sites that joined the initiative in 2016 provided self-reported employment data and self-assessed fidelity data. This included 34 outpatient mental health clinics operating in state psychiatric hospitals and 44 outpatient Personalized Recovery Oriented Services (PROS) community programs providing services aimed at promoting functional recovery for individuals with serious mental illness and operated by local non-profit agencies. Although both are found in the community, these two program types differed in organizational structure (clinic appointments vs. rehabilitation milieu) and governance (state government vs. non-profit agencies). The 78 community sites represent 67% of 51 state clinic sites and 48% of 92 PROS sites in the state in 2016. Of the 20 sites that were not included in this study, 6 sites were not providing IPS at the time and 14 did not provide fidelity assessments. It should be noted that there were no statistically significant differences in one month employment rates, and in geographic region of the state, between the 78 programs in this study and the 14 programs that were excluded because of the lack of fidelity assessments.

Site program staff (i.e., employment staff and supervisors) provided self-reported fidelity assessments, using the IPS Supported Employment Fidelity Scale (found at https://ipsworks.org/wp-content/uploads/2017/08/IPS-Fidelity-Scale-Eng1.pdf). The IPS learning collaborative initiative emphasizes fidelity. In the learning collaborative, program staff were taught about IPS fidelity generally, and how to conduct fidelity self-assessment specifically, through webinars and program-specific consultation calls and visits. They received 1 to 3 hours of specific fidelity training that was augmented by a focus on fidelity that was intertwined throughout the entire training experience each year (14). Fidelity self-assessments are completed annually by participating programs. In a recent study comparing self-assessed fidelity to independent ratings (14), we found agreement between these methods when looking at programs’ total Scale score, though there was variability in agreement for individual items.

Using an online survey via the Qualtrics software platform, participating programs provide monthly performance indicator data. This includes staffing allocations, caseload size, and competitive employment status of those receiving IPS. The employment data used in this study was reported for the month in which the fidelity assessment was conducted. During this month, all participating programs reported the number of people employed competitively, defined as community-based job that pays at least minimum wage, is available to any person (i.e., not created specifically for people with disabilities), may be full time or part time, and does not have time limits determined by the mental health agency. Each program also submitted the number of people on their caseload, defined as the number of individuals receiving at least one individual IPS service during the referenced time period (i.e., one or three months). One program type, PROS (n=44), also reported competitive employment in a 3-month period (including the month in which the fidelity measure was submitted and the 2 months prior to that) and provided 2 separate fidelity estimates (with baseline fidelity collected between December 2015 and June 2016, and follow-up fidelity collected in December 2016). We calculated a rate (percentage) of employment by dividing the number of people who were competitively employed (1- or 3-month) by the caseload.

To replicate the analyses in the two recent IPS Learning Community studies (10, 11), we used the Pearson correlation to examine our primary hypothesis, that total fidelity score would correlate positively with 1-month competitive employment rate (all sites), and 3-month competitive employment rate (PROS sites only). Secondarily, we examined correlations of individual fidelity items with 1-month competitive employment rate, using the Bonferroni correction to adjust the p-value for multiple comparisons (.05/25=.002). Within PROS programs, we conducted a mixed-effects model to examine the relationship between repeated fidelity and employment measures over time. Specifically, the mixed effect model takes into account the correlation between repeated measures when estimating the relationship between fidelity and employment measures. Analyses were conducted using IBM SPSS Statistics version 23.

Results

Across the 78 programs, total fidelity scores ranged from 58 to 122 (M = 93.76±12.80), with almost one-third of the programs (N=25) achieving good to high fidelity scores (> 99). The 1-month employment rate ranged from zero to a hundred percent (M = 43%±25). We found statistically significant correlations between total fidelity scores and 1-month employment rate, r = .331 (p = .003). In examining individual fidelity items using the Bonferroni correction, one item was significantly correlated with the 1-month employment rate, as noted in Table 1: diversity of jobs developed, r = .364 (p =.001).

Table 1.

Correlations Between Fidelity Assessments and Employment Rate (1-Month Competitive Employment Rate)^a

Item	Mean	SD	Correlation	p-value
Overall Fidelity Score	93.76	12.80	.331*	.003
S1 Caseload size	4.08	1.356	−.121	.292
S2 Vocational services staff	3.44	1.542	.270	.017
S3 Vocational generalists	4.19	0.884	.196	.086
O1 Integration of rehabilitation with mental health treatment through team assignment	4.54	1.028	.164	.151
O2 Integration of rehabilitation with mental health treatment through frequent team contact	4.65	0.735	.091	.430
O3 Executive team support for supported employment	2.13	1.273	.195	.087
O4 Vocational unit	2.54	1.411	.238	.036
O5 Role of employment supervisor	2.90	1.234	.135	.239
O6 Zero exclusion criteria	4.73	0.767	.132	.248
O7 Agency focus on competitive employment	4.50	0.679	.139	.225
O8 Executive team support for supported employment	3.28	1.347	.012	.916
SE1 Work incentives planning	4.18	1.137	.316	.005
SE2 Disclosure	4.56	0.594	.086	.456
SE3 Ongoing, work- based vocational assessment	4.17	0.973	.134	.243
SE4 Rapid search for competitive job	3.53	1.078	−.017	.884
SE5 Individualized job search	4.44	0.975	.197	.083
SE6 Job development: frequent employer contact	2.63	1.378	.140	.222
SE7 Job development: quality of employer contacts	3.67	1.393	.124	.281
SE8 Diversity of jobs developed	3.71	1.442	.364**	.001
SE9 Diversity of employers	4.00	1.173	.134	.241
SE10 Competitive jobs	4.09	1.219	.274	.015
SE11 Individualized follow-along supports	4.40	0.944	.219	.054
SE12 Follow along supports: time unlimited	3.92	1.042	.113	.326
SE13 Community- based services	2.00	1.151	.232	.041
SE14 Assertive engagement and outreach by integrated team	3.50	1.182	−.068	.557

^aCombined data from PROS and state clinics during the year 2016 (N=78)

^*Correlation is significant at the .01 level, 2-tailed.

^**Correlation is significant at the .002 level, 2-tailed.

In examining the 44 PROS programs with 3-month outcomes, baseline total fidelity score averaged 91.5±14.7 and 3-month employment rate averaged 38%±23, while follow-up total fidelity score averaged 92.2±14.5 and 3-month employment rate averaged 43%±23. Similar to 1-month employment outcomes, we found significant correlations between total fidelity scores and 3-month employment rate at both baseline, r = .366 (p = .015), and at follow-up, r = .378 (p = .025). In the mixed-effects model, examining this relationship over the two time points, total fidelity score remained a significant contributor, F (1, 67.662) = 4.010, p = .049 to the 3-month employment rate, though the results did not change significantly between the two time points.

Discussion

This study demonstrated a significant positive relationship between IPS self-reported fidelity and employment, consistent with prior studies assessing programs participating in the IPS Learning Community (10, 11) and with programs contracted by the Massachusetts DMH (12). We also found significant positive correlations between one individual fidelity item and employment outcomes. Our study extends this prior work in demonstrating that the relationship between fidelity and employment was sustained through time (up to one year). Additionally, this study shows that the relationship was observed in agency self-reported fidelity (rather than fidelity assessed by an external reviewer).

As the demand, particularly among federal and state payers, for evidence-based practices grows, so does the demand for fidelity assessments (15). This study not only provides additional confirmation for the hypothesis that high-fidelity programs have better employment outcomes, and shows that these effects are sustained over time, but also shows that programs supported by the NYS learning collaborative have shown similar employment rates as the original IPS Learning Community. This underlines the efficacy of the IPS model and its robustness to implementation by experts other than the original IPS developers. This replicability is key for demonstrating the overall efficacy of the IPS model. Our results also show the robustness of the model in two different program types (outpatient clinics operated by state psychiatric facilities, and recovery-oriented programs operated by independent, community-based non-profit agencies).

As in the studies of the IPS Learning Community (10, 11) we find that individual item(s) in the IPS fidelity assessment are significantly associated with employment, but the specific items differ among the three studies. This indicates the importance of utilizing the full fidelity instrument when assessing program quality indicators; no subset of items has been shown to consistently deliver comparable results.

The fidelity assessments in our study are self-reported by program sites; this can be considered both an asset and a limitation. While independent, external assessment is an important accountability measure, it can be expensive and burdensome for sponsoring agencies to provide on an ongoing basis (14). Our prior study showed that agency self-report can yield similar fidelity scores to independent assessors within the NYS learning collaborative (14) and this study’s results indicate that self-reported fidelity, like independently-assessed fidelity (10, 11, 12), is associated with greater employment rates.

This is an observational study that does not demonstrate causality. It is possible that programs who choose to invest in high-fidelity have other characteristics that contribute to high employment rates among their clients (for example, additional services may be offered that also contribute to employment rates).

Future research should look at whether site characteristics such as program size or longevity are related to fidelity and/or employment outcomes, examine why programs choose to invest in providing IPS services, identify the most significant fidelity items contributing to positive employment outcomes, consider employment outcomes beyond employment status such as job tenure and hours worked each week, and identify mediators and moderators at the client and site levels that may influence program outcomes.

Conclusions

This study presents further evidence about the importance of implementation fidelity for IPS supported employment. IPS fidelity self-assessed by program sites is shown to be associated with higher employment rates, which are sustained over time. Employment is a normative activity for individuals with serious mental illness and can be an important component of their recovery. Employment can also provide both an income and a quantifiable way to participate in the community. Programs that offer high-fidelity IPS services can help their clients achieve their goals.