Variation in the Implementation of California's Full Service Partnerships for Persons with Serious Mental Illness

Abstract

Objective. This study examined variation in the implementation of California's Full Service Partnerships (FSPs), which are supported housing programs that do “whatever it takes” to improve outcomes among persons with serious mental illness who are homeless or at risk of homelessness.

Data Sources/Setting. Ninety-three FSPs in California.

Study Design. A mixed methods approach was selected to develop a better understanding of the complexity of the FSP programs. The design structure was a combined explanatory and exploratory sequential design (qual→QUAN→qual) where a qualitative focus group was used to develop a quantitative survey that was followed by qualitative site visits. The survey was used to describe the breadth of variation based on fidelity to the Housing First model, while the site visits were used to provide a depth of information on high- versus low-fidelity programs.

Principal Findings. We found substantial variation in implementation among FSPs. Fidelity was particularly low along domains related with housing and service philosophy, indicating that many FSPs implemented a rich array of services but applied housing readiness requirements and did not adhere to consumer choice in housing.

Conclusions. There remains room for improvement in the recovery-orientation of FSPs. Fortunately, we have identified several processes by which program managers and counties can increase the fidelity of their programs.

On November 2, 2004, California voters approved Proposition 63, which was signed into law as the Mental Health Services Act (MHSA). The MHSA applied a tax of 1 percent on incomes over $1 million to fund public mental health services (Scheffler and Adams 2005). The cornerstone of the MHSA was the implementation of Full Services Partnerships (FSPs). FSPs are integrated supported housing and team-based treatment models that do “whatever it takes” to improve residential stability and mental health outcomes among persons with serious mental illness who are homeless or at risk of homelessness (Gilmer et al. 2010).

Consistent with prior efforts focused on reforming the delivery of mental health care in California, the MHSA emphasized concepts of integration, recovery-orientation, and flexible funding of services, as well as stakeholder engagement and community involvement (Cashin et al. 2008). FSPs would provide a range of services, including housing, mental health services, and educational, vocational, financial, and social supports. Services would be client-centered, building on the strengths and resiliencies of clients, and targeting long-term recovery and self-sufficiency by providing the physical, emotional, and intellectual skills needed by persons with mental illness to live, learn, and work in their particular environments. The FSPs were also expected to be responsive to stakeholders and adaptive to local contexts. The emphasis on the vision of integrated, recovery-oriented care that “does whatever it takes,” the flexibility in funding, and the influence of stakeholders, combined with a lack of specificity and oversight regarding expected FSP practices, led to the implementation of a diverse array of FSP programs (Cashin et al. 2008).

The goal of this research was to describe variation in the implementation of FSP programs and to relate specific components or causes underlying this variation to current constructs in implementation science (Damschroder et al. 2009). The Consolidated Framework for Implementation Research (CFIR) offers a overarching typology for implementation research and comprises five major domains: the intervention, the inner and outer settings in which it is implemented, the individuals involved in implementation, and the process by which implementation is accomplished. A better understanding of the variation in implementation of the FSPs may inform whether the overarching philosophy of the MHSA allowed FSPs to implement recovery-oriented programs that were optimized for individuals needs, or whether the lack of specificity and oversight resulted in variation that reflected a mix of recovery-oriented and traditional models of care. Relating this variation to the CFIR domains may inform the development and dissemination of strategies to optimize the effectiveness and sustainability of FSPs in specific contexts, and it may also lead to the identification or more precise specification of FSP best practices.

Since there was no existing conceptual framework to describe FSP practices, a framework was developed to compare FSP practices to a benchmark program sharing similar goals, vision, and structure: the Housing First Model. Housing First programs provide immediate access to affordable, permanent, scattered-site housing with tenancy rights, and team-based services according to a recovery-oriented service philosophy, which draws heavily on the psychosocial rehabilitation model (Tsemberis 1999). Key elements of this approach are consumer choice, self-determination, and independence; the active use of harm reduction, motivational interviewing, assertive engagement, and person-centered planning by program staff; and the absence of coercive practices.

Studies of Housing First have found it to be effective at improving residential outcomes among homeless persons with SMI (Tsemberis and Eisenberg 2000; Tsemberis et al. 2003; Tsemberis, Gulcur, and Nakae 2004; Mares and Rosenheck 2007; Pearson et al. 2007). The Substance Abuse and Mental Health Services Administration considers the Housing First model to be an evidence-based practice (SAMHSA 2007). Pathways to Housing, Inc. has developed a site-visit-based fidelity scale that can be used to evaluate the consistency of supported housing programs with the Housing First model (Tsemberis 2010). In this sequential mixed methods study, we develop a quantitative self-administered survey-based measure of fidelity to Housing First, which we administer across a diverse array of FSP programs, and which we use in conjunction with qualitative site visits to examine variation in implementation of California's FSPs.

Methods

Design

A sequential mixed methods approach was selected to assess intervention fidelity and to develop a better understanding of the complexity of the FSP programs and the multicomponent services that they provide (Alexander and Hearld 2012). The combined exploratory and explanatory sequential design was qual→QUAN→qual, where a qualitative focus group preceded a quantitative survey that was followed by qualitative site visits (Palinkas et al. 2011a; Fetters et al. 2013, this issue). The design incorporated five distinct functions of integrating quantitative and qualitative data: development, sampling, convergence, complementarity, and expansion (Table 1) (Palinkas et al. 2011b). A focus group was used to assist in the development of a survey-based measure of fidelity to the Housing First model (Fetters et al. refer to this as bridging). This survey was then administered across a large sample of FSP programs serving adults, adults exiting the justice system, transitional age youth, and older adults. Results from the survey were used to purposefully sample 20 programs for site visits (Fetter et al. refer to this as connecting). The site visits involved the collection of qualitative data through direct observation, interviews with program managers and staff, and focus groups with clients, and assignment of a fidelity score. The survey and site visit data were combined to facilitate interpretation of the data through mechanisms of convergence (comparing fidelity scores derived from the survey to those derived from the site visits), complementarity (using the quantitative survey data to provide a breadth of information and the qualitative site visit data to provide a depth of understanding), and expansion (using the qualitative site visit data to help understand and explain the results from the analyses of the quantitative survey data).

Table 1.

Characteristics of the Mixed Methods Design for Studying Variations in Implementation among Full Service Partnerships

Function	Description and Sample	Goal
Development	An expert focus group was used to develop a survey based version of the Housing First fidelity scale	To develop a self-administered, survey-based measure of fidelity to the Housing First model
Sampling	Survey data was used to select 20 programs for site visits from the 93 programs surveyed	To purposefully sample programs to provide rich data on variation in implementation
Convergence	Fidelity scores derived from the survey were compared to fidelity scores derived from the site visits for 20 programs with site visit data	To determine the extent to which assessments of fidelity are consistent when measured by survey versus site visit
Complementarity	Survey data were used to select high-and low-fidelity programs that were compared using qualitative data from the site visits	To provide both a breadth and a depth of understanding of variations in implementation
Expansion	Survey data identified county variations in fidelity that were further explored using qualitative data from the site visits	To use qualitative data to help understand and explain results from the quantitative data

Procedure

Development of the Housing First Fidelity Survey

The Housing First fidelity scale was developed for use during a site visit with multiple independent raters, which was not feasible to conduct across a large number of programs. Therefore, it was necessary to adapt the fidelity scale to a survey format. Fidelity items that were scored by trained evaluators after direct observation were reconfigured to allow responses from program staff. A focus group reviewed the survey questions for relevance to fidelity, relevance to the FSP programs, the objectivity of the question, and the ability of FSPs to respond. Ultimately, 46 questions were approved to measure fidelity across the five domains (Tsemberis 2010): (1) housing choice and structure; (2) separation of housing and services; (3) service philosophy; (4) service array; and (5) program structure. Additional information on survey administration and fidelity scoring using factor analysis is provided elsewhere (Gilmer, Stefancic, and Sklar 2013).

Sampling FSPs for Site Visits

A map of participating counties is shown in Figure 1. Twenty-three counties participated in the survey, including counties in northern, central, and southern California, coastal and inland counties, and counties with large urban and rural populations (many rural counties with smaller populations did not receive enough funding to support FSP programs). Among participating counties, 93 of 135 FSP programs (69 percent) responded to the survey.

Figure 1.

Counties Participating in the Full Service Partnership Survey

Quantitative results from the survey were used to purposefully sample 20 FSP programs for site visits. Sampling was a two-stage process. At the first stage, counties were purposefully sampled using a maximum variation strategy (Patton, 2002) to provide geographic, political, and economic diversity, as well as a wide range of fidelity scores. Participating counties were selected from southern, central, and northern California, both coastal and inland, and urban and rural. Fidelity scores exhibited some clustering by county, and therefore counties were selected that might provide some insight into the reasons behind this clustering. Within counties, programs were selected to maximize the range of fidelity scores, as well as to provide representation of programs designed for specific populations, including adults exiting the justice system, older adults, and transitional age youth. All of the programs that were selected agreed to participate in the site visit.

Convergence

Twenty programs that responded to the survey were selected for site visits, which were conducted by an experienced team from Pathways to Housing, Inc. and a qualitative researcher [M. L. K.]. The purpose of the site visits was to acquire an in-depth understanding of the FSP programs' fidelity to the Housing First model through direct observation, interviews with staff, focus groups with clients, and chart abstracts. Interviews and focus groups were recorded, providing qualitative data. The FSP programs were scored on each of the five fidelity domains. These scores, based on a qualitative understanding of the programs, were compared with the scores obtained from the quantitative fidelity survey. Convergence involved merging these data, and determining the extent to which fidelity scores derived from the survey were correlated with fidelity scores derived during the site visits.

Complementarity

The survey and site visit data were connected to provide complementarity; the quantitative survey provided a breath of information on fidelity across study sites, while the qualitative site visits provided a depth of understanding of the experience of FSP implementation within a specific site. The survey was used to identify the top five and bottom five programs with respect to fidelity. The qualitative analysis of variation in fidelity focused on identifying important elements of these high- and low-fidelity programs and the extent to which these elements reflected the domains for the CFIR model based on data from the FSP program director interviews.

Expansion

The qualitative data were next used to answer questions that emerged during the process of analysis of the quantitative data. For example, as described below, the survey data identified county-level patterns of fidelity to the housing first model, but they could not be used to explain these patterns. Qualitative analysis of the FSP program director interviews, however, enabled us to begin to understand and explain the county-level variation that was identified in the survey data.

Qualitative Methods

Interview transcripts were analyzed using a methodology of “Coding Consensus, Co-occurrence, and Comparison” outlined by Willms et al. (high- and low-fidelity) (1990) and rooted in grounded theory (i.e., theory derived from data and then illustrated by characteristic examples of data) (Glaser and Strauss 1967). The second author coded the program director interview transcripts using a master codebook created through an iterative, collaborative process involving the fourth author and two research assistants. To create the codebook, the second and fourth authors each separately conducted an initial round of open coding on a set of randomly chosen transcripts. The lists of codes developed by each investigator were matched and integrated into a single master codebook, which included short descriptive statements and examples to illustrate and define the boundaries of specific codes (i.e., the inclusion and exclusion criteria for assigning a specific code) (Miles and Huberman 1994). Next, a new sample of transcripts was independently coded by the second and fourth authors and the two research assistants. Disagreements in assignment or description of codes were resolved through discussion among all four investigators, resulting in enhanced definitions of codes. The final codebook consisted of a numbered list of categories representing themes, issues, accounts of behaviors, and opinions relating to individual, organizational, and system characteristics that influence implementation of the FSP model. With the final coding structure, the four investigators separately reviewed a random sample of transcripts to determine level of agreement in the codes applied. A level of agreement in the codes applied ranged from 66 to 97 percent depending on level of coding (general, intermediate, specific), indicating good reliability in qualitative research (Boyatzis 1998). Based on these codes, the computer program Atlas.ti (Muhr 2004) was used to selectively code the program director interviews for segments of talk relevant to the issues of county influence and high and low fidelity. The Atlas.ti program was further used to sort the coded segments and explore relationships among the categories, and to create new codes for categories that emerged during the coding process, that is, to identify the existence of new, previously unrecognized categories. Through the process of constantly comparing these categories with each other, the categories of interest were identified and further condensed into broad themes that were summarized and used to complement and expand on the quantitative findings (Palinkas et al. 2011a). Finally, a template approach (Crabtree and Miller 1992) to text analysis was employed to create a two-dimensional matrix with high- and low-fidelity programs on one dimension and the list of CFIR predictor variables on the other dimension. This template enabled us to make two types of comparisons between high- and low-fidelity programs: (1) presence or absence of a specific CFIR variable in the identified codes/themes elicited and (2) contrasts in the character of the CFIR (categorical or interval) if present in both groups of programs.

Results

Quantitative Findings

Few FSP programs reported high fidelity to Housing First ingredients as measured by several items in the housing/service philosophy domain (see Table 2): only 14 percent of programs indicated that at least 85 percent of their participants were living in scatter-site housing, only 43 percent of programs reported not have housing readiness requirements such as completion of time in transitional housing or treatment, sobriety or abstinence, medication compliance, psychiatric symptom stability, or willingness to comply with a treatment plan; and only 30 percent of programs provided standard lease agreements without similar treatment provisions.

Table 2.

Fidelity to the Housing First Model among Full Service Partnerships Responding to the Housing First Fidelity Survey (N = 93)

Housing/Service Philosophy	High Fidelity (%)
Housing choice and structure
Fewer than 30% of participants live in emergency, short-term, transitional, or time-limited housing	73
At least 85% of participants live in scattered-site permanent supported housing	14
Separation of housing and services
Access to permanent housing requires only face-to-face visits with program staff and adhering to a standard lease	43
The majority of participants in permanent housing have a lease or occupancy agreement that specifies their rights and responsibilities of tenancy and which do not include provisions regarding adherence to medication, sobriety, or a treatment plans, or adherence to program rules such as curfews or restrictions on overnight guests	36
Service philosophy
Participants have the right to choose, modify, or refuse services and supports at any time	63
Participants with serious mental illness are not required to take medication and/or participate in treatment	67
Participants with substance use disorders are not required to participate in substance use treatment	81
Program follows a harm reduction approach to substance use	76

Service Array/Program Structure	High Fidelity (%)
Service array
Program provides three or more approaches to substance use intervention	69
Program provides opportunities for community-based employment	75
Program provides opportunities for supported education in the community	88
Program provides opportunities for community-based volunteering	93
Program provides three or more approaches to support participants with physical health issues	71
Program provides three core social integration services	71
Program structure
Program staff meets at least 4 days a week	41
Program meetings address four core functions	74

In contrast, more FSP programs met the standards for high fidelity in the service array/program structure domain, including not having requirements for participation in services (63 percent), pharmacotherapy (67 percent), or substance use treatment (81 percent). Similarly, the majority of programs endorsed a harm reduction approach to substance use (76 percent). Most programs also met fidelity standards with respect to availability of substance use services as well as services to support community employment, education, volunteering, physical health care, and social integration. Most also utilized a team approach to service delivery wherein participants received services from multiple staff.

Merging the survey and site visit data allowed comparison of fidelity scores among the subset of 20 programs where site visits were conducted. The housing/service philosophy factor from the survey was strongly correlated with the fidelity score from the site visit: ρ = .708, p < .001. The service array/program structure factor was moderately correlated with the fidelity score from the site visit: ρ = .335, p = .149. Visual comparison of the scores obtained from the two methods revealed that there were two programs that scored high on the service array/program structure factor in the survey but low on the site visit fidelity score. Excluding these two programs increased the correlation between the service array/program structure factor and the site visit fidelity score: ρ = .635, p = .005.

The survey was used to identify the top five and bottom five programs with respect to fidelity housing/service philosophy, and the top five and bottom five programs with respect to service array/program structure. The resulting 15 programs were studied in more depth using qualitative analyses of the program manager interviews: eight programs were high fidelity with respect to housing/service philosophy and/or service array/program structure, and seven programs were low fidelity in one or both factors. The survey was also used to identify county effects on fidelity to housing/service philosophy: the interclass correlation coefficient was .88 (SE = .08), compared with .25 (SE = .27) for service array/program structure. Finally, the survey was used to examine variations by program type. FSPs that served adults exiting the criminal justice system had lower fidelity to housing/service philosophy (ρ = −.23, p = .029), resulting from court-mandated requirements for participants to complete time in transitional housing or treatment before gaining access to permanent housing: 50 percent of criminal justice FSPs had such requirements compared with 23 percent of other FSPs (p = .034).

Qualitative Findings

Analysis of transcripts of the program directors' interviews revealed 15 qualitative themes that were consistent with three of the five major domains of the CFIR (Damschroder et al. 2009): characteristics of individuals (in this case, the program directors), inner setting, and outer setting (see Table 3). These themes are described below.

Table 3.

Qualitative Themes and Their Related Consolidated Framework for Implementation Research Domains

Implementation Domains	Theme	High Fidelity	Low Fidelity
Characteristics of individuals (Program Director)
Knowledge and beliefsabout intervention	Value orientation to full service partnership (FSP) goals	“As executive director…[I] try to enhance the environment so that it's consistent with recovery values.”	“…it's a crisis oriented team because we're always putting out fires.”
Knowledge and beliefs about intervention	Prior experience with programs similar to high fidelity FSPs	“…as part of my fourth year elective [as a Psych resident], I worked half time for the whole year at The Village…and I completely became converted to the cause of psychosocial rehab…”	“…my background has been in corrections…and I've only been at this position for the last 8 months. And so it's a huge transition for me, because in jail it was predominantly a medical model, and coming here it was completely different.”
Knowledge and beliefs about intervention	Value orientation to client	“I spend a lot of time helping people [staff] look at the perspective of the member [client], and then helping them move that way instead of what we think is the best thing for them…”	“…what we're really saying is, ‘How do I see something from their perspective to get their buy-in and reframe it?’ That's all it is is reframing it in a way that's digestible and palatable for them. And so, yeah it's manipulation, yes, but we believe that we're doing it with the best intentions.”
Knowledge and beliefs about intervention	Medication philosophy	“I believe that if we help them get on SSI and help them get housing that their mental health will improve dramatically. Forget the medications, because there's no meds for poverty, and there's no meds for homelessness, and there's no meds for childhood trauma either, which we have a lot of.”	“…we've saved a tremendous amount of money [on] client medications and client housing because we have a lot of other options…we try to come up with really sustainable resources for folks, and so we don't spend nearly what our budget amount is.”
Knowledge and beliefs about the intervention	Housing philosophy	“I lean more toward the scattered site…when people start talking about site-based services, what the hell is normalizing about that?”	“…there are plenty of people that don't really want any kind of housing. They'd rather just bounce around in hotels, and that's easier and better for them.”
Other personal attributes	Political awareness	“…they [city leaders] said, ‘Well, this is an older section of ((city))…And property values have dropped in this area … so we'll create this containment area. And it conjures up for me old Europe and Jewish ghettos…yet at the same time you have people running programs down here in the area [saying] ‘Well, let's build another hotel down there.’”
Inner setting
Culture	Program goals	“…people are people. We're here to help them in their quality of life and to be what they want to be.”	“Our main goal is really to keep them from going to jail and from getting back in the hospital.”
Culture; compatibility	Continuity or change in approach to housing and services	“…the agency has always been very focused on the emerging best practices for folks with co-occurring disorders, the evidence-based practices for folks with severe and persistent mental illness…we've operated housing now for over thirty years as an organization…”	“We didn't really have any structure for how to use them [flex funds], and we were hearing this ‘whatever it takes’ and ‘housing first’ and not really knowing how to do that…”
Compatibility (also related to Characteristics of individuals, Knowledge and Beliefs about the Intervention; and Process, Engaging)	Staffing	“…it's constant in this agency, the trainings on it [recovery],…and when you hire people you look for it, you know, ‘what is your position on it? Can the guy who is using crack have a job? What do you think?’ ‘Absolutely not; you've got to be abstinent.’ Cool, you don't need to work here.”	“…I think that we have to work with the staff that we have in terms of their comfort level…their capacity and their skills… And it really is generally kind of hard with their days that are so full of delivering direct services…I mean people have to do what they feel confident with, so if there's not gonna be some intensive kind of training, then the changes, I think, have to be more gradual.”
Culture	Control of decision making	“…there are so many issues around where in the county somebody feels comfortable to be. What neighborhood, the size of the building. Some people do better when there is a gate out front…”	“The team first starts to decide where we think clients might do well [re: housing]… So we kind of made the decision for her to put her in an ILF [Independent Living Facility].”
Networks and communication; culture	Language/usage	“…language is the primary way we communicate with people…and knowing that you convey a different level of respect…ultimately gaining a different level of effectiveness by the language you use, by the way you conceptualize, through your language, a cooperative effort in terms of services. So, it becomes services done with somebody not to somebody…”	“I think writing that way [writing for Medi-Cal using recovery language] really helps you…start to think that way, you know, cognitively… and like I said I'm trying—I mean, we're getting there. But I think that that would help not being so rigid and do this medical model.”
Readiness for implementation	Distribution of housing funds to clients	“We have money for housing…so anybody who needs it…we pay for their housing until we can help them establish that.”	“I have maybe eleven or twelve clients that we pay for, otherwise they are with family or they are with SSI and/or we're just paying a supplemental rate because they are at a high level of Board and Care. And so SSI will cover them to a point and then we just have to pay the little supplemental rate.”
Outer setting
Patient needs and resources	Target population	“We do outreach, and we find people in the community that are unserved because part of our population is- you are supposed to be unserved by the mental health system. So there's no list that the county has of who those folks are because [they are] the folks that the county hasn't served.”	“Here in [County Name] the greatest need has been high utilizers of hospitals and jails, so I would say at least 85 percent of our referrals are coming from those settings.”
Cosmopolitanism	FSP networks and social capital	“They [the Village] refer to us a lot. like [name of Village ED] will call up and say “Hey, I've got some older adults for you…so he'll actually refer people to us and try to get people into our program because he knows that we're good…and our administrator's been around for a very long time and he knows the president over there [at the Village], so we're just constantly in contact…”	“…it's also a county thing is they're really rigid in terms of billing to Medi-Cal standards, which is more a traditional medical model. And so it's gonna be hard when you're reading our charts to really pick up on us being strength based or being stage based… And I've heard, at these FSP network meetings, other agencies say that they've been able to kind of get around that and been able to document more to that style. I'm not sure how they do it or how they go about it.”
External policy and incentives	County policies affecting FSP implementation	“…we had a lot of guidance from the county…there were trainings for how to do an ACT [Assertive Community Treatment] model, how to do an FSP model, and we continue to get trainings from the county frequently.”	“They [the county] don't want any of the programs to have a wait list…So that creates a challenge for us because on the one hand we have to make sure we're providing service to our clients, and with the level of functioning in a lot of our clients it can get challenging to determine who we move on so we can make room for other clients coming into the program.”

Individual Characteristics

This CFIR domain captures the role of individuals involved with the intervention and/or implementation process, with the understanding that “individuals are carriers of cultural, organizational, professional, and individual mindsets, norms, interests, and affiliations.” Interventions are implemented by individuals, and thus it is important to understand how individual characteristics affect the implementation process. Six themes can be described as individual characteristics of program directors: (1) value orientation to FSP goals; (2) prior experience with programs similar to high-fidelity FSPs; (3) value orientation to the client; (4) medication philosophy; (5) housing philosophy; and (6) political awareness and/or connections.

Value orientation to FSP goals: Directors of high-fidelity programs display values-based leadership focused unequivocally on client needs, which tends to segue explicitly or implicitly with a recovery orientation. In contrast, directors of low-fidelity programs emphasize non-recovery-oriented and non-client-focused priorities, such as achieving cost savings, and treat as acceptable or even endorse physical settings (e.g., restricted client access to staff offices; guarded entryways) and behavioral norms (e.g., crisis orientation) that hinder or contradict recovery.

Prior experience with programs similar to high-fidelity FSPs: The prior work experience of program directors is associated with fidelity, with directors of high-fidelity programs more likely to have had significant experience working in similar service models with a psychosocial rehabilitation or recovery approach. Directors of low-fidelity programs were more likely to have experience in more traditional models of care.

Value orientation to client: Directors of high-fidelity programs orient to clients as equals. They tend be empathic with client perspectives and experiences and treat clients' points of view as valid and authoritative, and they expect their staff to do the same. Directors of low-fidelity programs orient to clients as less competent than themselves and their staff. They tend to be paternalistic and feel justified in making decisions for clients, even when those decisions are different from client preferences.

Value orientation to treatment—medication: This theme related to the importance of medication and its role in the treatment process. In the contrast presented in Table 3, the director of a high-fidelity program prioritizes income and housing over medication as most important for clients' mental health, while the director of a low-fidelity program prioritizes saving money and does not distinguish between medication and housing in terms of treatment effectiveness.

Value orientation to treatment—housing: The high-fidelity program directors were not equally emphatic about scattered site housing, but they shared a philosophy of housing as a treatment intervention, preferring scattered site housing with an understanding of its therapeutic value. Directors of low-fidelity programs were more likely to display an ad hoc pragmatic view toward housing as getting people off the streets and out of institutions, but without a sense of the different clinical or recovery implications and challenges of different housing options.

Political awareness/connections: Four of the high-fidelity program directors displayed a high level of awareness of the local (city and/or county) political context and how it affected their FSP programs. Two of these individuals had also been active in trying to influence the political context in terms of shaping policy. In contrast, none of the low-fidelity program directors discussed how their programs fit into or were affected by local politics.

Inner Setting

Damschroder et al. (2009) consider that “the line between inner and outer setting is not always clear and the interface is dynamic and sometimes precarious.” Specific characteristics of inner setting include “tangible and intangible manifestation of structural characteristics, networks and communications, culture, climate, and readiness.” For our purposes, inner setting refers to aspects of the FSP programs and includes any larger program or agency that directly influence program culture, staff, policies, or protocols. The six themes identified in the program director interviews that relate to inner setting cluster around culture, climate, and readiness. They are (1) program goals; (2) continuity or change in approach to housing services; (3) staffing; (4) control over decision making; (5) language/usage; and (6) distribution of housing funds to clients.

Program goals: Program directors spoke generally about the program goals in ways that provide insight into the culture of their programs because they embody organizational norms, values, and basic assumptions. Whereas directors of high-fidelity programs emphasize that the goal of the program is to improve clients' quality of life, directors of low-fidelity programs emphasize managing client service utilization, such as keeping them out of hospitals and prisons.

Continuity or change in approach to housing services: Continuity captures whether the FSP philosophy was largely consistent with the existing staff's orientation and program structure. All high-fidelity programs that were in existence prior to becoming FSPs had a service philosophy and approach to services that closely matched the design of the FSP. In contrast, low-fidelity programs were operating according to a traditional philosophy and continuum of care approach, and would have required a great deal of change to be brought into alignment with the recovery-oriented FSP philosophy.

Staffing: High-fidelity programs place great importance on hiring staff whose personal values and beliefs align with recovery-oriented goals and approaches. High-fidelity program directors with less control over hiring may identify staffing as a problem or describe actions they have taken to encourage and support staff to move in a recovery-oriented direction. Low-fidelity program directors often have little control over staffing, as illustrated in Table 3. In cases where the low-fidelity director does have control over hiring, a recovery orientation does not seem to be a central criterion.

Control over decision making: Program directors' descriptions of decision making on behalf of clients reveal whether they and their programs are committed to client-driven or client-centered decision making. Directors of high-fidelity programs describe client-centered approaches to decision making, which involve clients as active participants in decision making and a staff orientation to taking clients' expressed preferences seriously. Directors of low-fidelity programs describe decisions being made for clients by staff with little involvement of clients themselves and with little weight given to client preferences.

Language/usage: Several high-fidelity program directors made explicit reference to the importance of language for implementing client-centered and recovery-oriented services. Low-fidelity program directors displayed less concern with language per se, and often used language that was at odds with a recovery orientation. In cases where low-fidelity program directors were aware of the importance of language, they describe barriers to implementing the use of recovery-oriented language among staff.

Distribution of housing funds to clients: Program directors had different approaches to distributing housing subsidies to clients. This difference was striking since housing subsidies was a core feature of the FSP design. High-fidelity programs seemed to align with the understanding that their goal was to use the funds they had for housing to subsidize clients in permanent housing of the client's choice. Directors of low-fidelity programs expressed ambivalence about providing housing subsidies. This was illustrated in statements like “We're not a housing program” and through reports that they only pay for housing for a few of their clients.

Outer Setting

Damschroder et al. (2009) define outer setting as “the economic, political, and social context within which an organization resides.” The final three themes related to the outer setting of the programs: (1) target population; (2) FSP external networks and social capital; and (3) county policies.

Population served: Program directors of high-fidelity programs describe their clients as underserved. In contrast, directors of low-fidelity programs refer to their clients as “high utilizers” of services, meaning people who have frequent and/or prolonged contact with the hospital and prison systems. While these individuals may require additional assistance to avoid homelessness and maintain independence, they are nevertheless considered to be already engaged with the system and receiving services. When programs do outreach and recruit clients who have had little or no prior contact with the system, they are reaching the population for which programs like the FSPs and Housing First were primarily designed. While these distinctions are not absolute, since many programs do include individuals from both the unserved and high utilizing populations, they do indicate different emphases and understandings about the program purpose, which translate into different fidelity rankings.

FSP external networks and social capital: There is some evidence of larger and more extensive networks among high-fidelity programs. Moreover, having a high-fidelity program that is a “model” program may be associated with higher fidelity county-wide. The data suggest two possible mechanisms for this: the influence of model programs/activist directors on county policies and proximity of model programs enabling closer and more frequent contacts. Low-fidelity programs appear to be less well networked with high-fidelity programs. They may be networked with other low-fidelity programs, or they may not be aware of connections that would help them to transition to a higher fidelity program.

County policies: Program directors spoke at times explicitly about how county policies affected the implementation of their FSPs. These policies can promote or hinder fidelity. Many of the high-fidelity programs were already organized and functioning according to models very similar to the FSPs, but others described getting a great deal of support, guidance, and monitoring from the county in how to operate according to the FSP philosophy. In contrast, directors of low-fidelity programs described county policies that interfered with recovery goals. One of the most striking was one county's requirement that FSPs not have waiting lists for clients. This had the effect of imposing time limits on clients irrespective of their needs. According one director, it was often very difficult to decide who to discharge from the program, and discharged clients were likely to return. This requirement had wide-ranging effects on the program, including reconsidering program goals and per capita spending. All of the FSPs in this county were ranked as low-fidelity programs.

Discussion

This article examined variation in the implementation of California's FSPs. A sequential mixed methods approach was selected because of the absence of a conceptual framework for FSP programs, because of their utility in assessing intervention fidelity and developing a better understanding of the complexity of the FSP programs and the multicomponent services that they provide (Palinkas et al. 2011a; Alexander and Hearld 2012). We found substantial variation among programs in fidelity to the Housing First model. Fidelity was particularly low along domains related with housing and service philosophy, indicating that many FSPs implemented a rich array of services but applied housing readiness requirements and did not adhere to consumer choice in housing. The infusion of FSP funding may have served to expand existing resources for housing and services, but in many cases this expansion did not necessarily include adoption of a new program model or service philosophy. Rather, the funding enhanced programs' abilities to utilize existing networks of housing providers, which had most commonly been congregate/residential treatment settings such as room and board or board and care.

Our analysis of semistructured interviews with program directors revealed 15 themes organized into three domains of the CFIR (Damschroder et al. 2009): individual characteristics of program directors, the inner setting, and the outer setting. Directors of high-fidelity programs consistently put client recovery at the center of decision making and service provision. In contrast, directors of low-fidelity programs often put cost saving or other administrative concerns and staff judgments at the center of decision making. Thus, consistent with previous studies of the role of organizational leadership in implementation of evidence-based practices (Edmondson 2004; Aarons 2006), program director priorities and the underlying values or concerns determining the choice of priorities appear to be key determinants of program fidelity. Values are central to culture. As such, program directors may be theorized as important agents in the production, maintenance, and development of program cultures and their recovery orientation. Also consistent with previous studies of evidence-based practice implementation (Glisson and James 2002; Gershon et al. 2004; Greenhalgh et al. 2004; Glisson and Green 2006) is the finding that high-fidelity programs are more likely than low-fidelity programs to have cultures and implementation climates that are compatible with the housing first model and to display readiness for implementation. Finally, three aspects of the outer setting associated with program fidelity are the target population, connections with other FSPs, and county policies. Being located in a county with a model program appears to be a good predictor of fidelity, and this is supported by evidence of networking between model programs and other programs in the same counties.

This study had some limitations. Participation in the survey was voluntary, and not all FSPs participated. Although we developed a framework with input from FSP stakeholders, it is possible that we are missing important elements of the FSP model as conceptualized by those who implemented the MHSA legislation. The qualitative analyses employ data from only the FSP program director interviews. Future analyses will include data from interviews with other program staff. Despite these limitations, we believe that the findings of this study will inform both FSP practices in California as well as efforts aimed at implementing Housing First programs nationwide. Overall, we found that there remains room for improvement in the recovery orientation of some FSP programs, and we have identified several processes by which program managers and counties can increase the fidelity of their programs.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.