Implementation and Fidelity Assessment of the NAVIGATE Treatment Program for First Episode Psychosis in a Multi-site Study

Abstract

The NAVIGATE program was developed for the Recovery After Initial Schizophrenia Episode-Early Treatment Program (RAISE-ETP) study, which compared NAVIGATE to usual Community Care in a cluster randomized design involving 34 sites and 404 patients. This article describes the approach to training and implementing the NAVIGATE program at the 17 sites (including 134 practitioners) randomized to provide it, and to evaluating the fidelity of service delivery to the NAVIGATE model. Fidelity was evaluated to five different components of the program, all of which were standardized in manuals in advance of implementation. The components included four interventions (Individualized Resiliency Training, Family Education Program, Supported Employment and Education, Personalized Medication Management) and the overall organization (staffing and structure) of the NAVIGATE team. Most of the sites demonstrated acceptable or higher levels of fidelity in their implementation of the four interventions and the organization of the program, with all 17 sites demonstrating at least acceptable overall fidelity to the NAVIGATE program. The results indicate that the NAVIGATE program can be implemented with good fidelity to the treatment model in a diverse array of community mental health care settings serving persons with a first episode psychosis.

1. Introduction

Research over the past two decades has demonstrated beneficial effects of comprehensive treatment programs for people who have recently experienced a first episode of psychosis (FEP) (McGorry, 2015). This accumulation of evidence has led to efforts to identify the essential components of effective FEP treatment (Addington et al., 2013), referred to in the U.S. as Coordinated Specialty Care (CSC) programs (Heinssen et al., 2014). However, until recently most of research on the effectiveness of FEP treatment programs has been conducted in countries with universal (or near universal) healthcare and single payer healthcare systems such as Australia, Canada, and Europe, limiting the generalizability of the findings to the U.S. healthcare system (Kane et al., 2015).

To address the gap in knowledge about effective treatments for FEP that can be implemented in the U.S. healthcare system, the National Institute of Mental Health developed the Recovery After an Initial Schizophrenia Episode (RAISE) initiative, and issued a request for research proposals in 2008. These proposals were required to develop, pilot test, and rigorously evaluate FEP treatment programs that were tailored to the unique needs of this population, and could be implemented in typical “real world” treatment settings in the U.S., as well as be funded through existing payment mechanisms. The RAISE Early Treatment Program (ETP) was funded by this initiative. RAISE-ETP developed a CSC program for FEP and evaluated it in a cluster randomized controlled trial (Kane et al., 2016; Kane et al., 2015; Mueser et al., 2015).

The treatment that was developed and evaluated in the RAISE-ETP project is called NAVIGATE. The program was named “NAVIGATE” in order to convey its goal of helping clients and their family members find their way through the confusing experience of an FEP and the complexities of the mental health system towards the desired outcome of recovery. As a standardized, team-based program, NAVIGATE was designed to be implemented by existing staff in community mental health centers serving persons with FEP. The program was compared to usual Community Care in a cluster randomized controlled trial with a two-year treatment and follow-up. Primary analyses showed that clients in NAVIGATE had significantly greater reductions in overall psychiatric symptoms and depression, and greater improvements in quality of life, social relationships, and involvement in work and school compared to those who received usual Community Care (Kane et al., 2016).

The results provide strong evidence for the effectiveness of the NAVIGATE program in typical mental health treatment settings in the U.S. However, information has not yet been published on the methods used to implement NAVIGATE and to monitor site adherence to the treatment model, nor of the extent to which sites were able to successfully deliver the program. The systematic evaluation of clinicians’ (or program’) adherence and competence (or fidelity) at providing an intervention is important for several reasons.

First, increasing attention has focused on improving access to empirically supported practices for people with severe mental illness. Fundamental to this trend has been a widespread recognition of the importance of establishing that interventions shown to be effective in rigorous research trials can be implemented into routine service settings with acceptable levels of adherence to the principles of the treatment model (Addington et al., 2018 Ahead of Print). Effective interventions can only be expected to produce positive outcomes in typical treatment settings if they are provided competently and with good adherence to the original treatment model, and there is evidence linking better intervention fidelity to better clinical outcomes (Bond et al., 2011; McHugo et al., 1999).

Second, fidelity assessment can play an important role in training and quality improvement of services when the results are fed back to individual clinicians, supervisors, and teams. For example, the timely provision of fidelity feedback based on reviews of audio-recorded therapy sessions has been used to train clinicians in the cognitive behavioral treatment of posttraumatic stress disorder in people with severe mental illness, with accompanying client level evidence of improvement in targeted symptoms (Lu et al., 2012). Similarly, in the National Implementing Evidence Based Practices project, in which 53 mental health agencies each implemented two of five practices, fidelity assessments were conducted for each practice every six months, based on site visits, for two years, with the results reviewed with sites and incorporated into technical assistance plans for improving fidelity (McHugo et al., 2007).

Third, in a cluster randomized controlled trial design such as the RAISE-ETP project, the demonstration of a treatment effect favoring the experimental intervention does not address the question of whether different sites were in fact implementing the same intervention. This is a critical question because it informs the field as to whether the research supports the effectiveness of a specific treatment model (e.g., the NAVIGATE program) or a more general approach (e.g., having designated teams of clinicians provide treatment to people with an FEP). Such information has implications for the training and dissemination of programs based on research findings, and whether efforts should focus on implementing the defining components of an intervention or on the more general principles that the program is based on. For these reasons, this article focuses on describing the methods used to train clinicians in implementing the NAVIGATE program, and to evaluate their fidelity to the intervention.

2. Methods

Thirty-four community mental health agencies across 21 states were randomized to deliver either NAVIGATE (N=17 sites) or Community Care (N=17 sites) to persons with a first episode of nonaffective psychosis. A total of 404 clients aged 15–40 were recruited and assessed at baseline and every six months for two years by clinical interviewers who were masked to treatment assignment (Kane et al., 2016; Kane et al., 2015).

2.1. Study Sites

Potential study sites volunteered to participate after being made aware of the project and site requirements, which included: a) experience in treating schizophrenia; b) interest in offering early intervention services for FEP clients; c) sufficient staff to implement the NAVIGATE program; d) anticipated subject flow to recruit approximately 20 clients with FEP over a two-year period; and e) institutional assurance that research assessments would be completed. Exclusion criteria for sites were: a) academic programs (centers with academic affiliations were allowed); and b) centers with pre-existing specialty FEP programs. Descriptive characteristics collected in 2009 of the 17 sites randomized to provide the NAVIGATE program are provided in Table 1.

Table 1:

Characteristics of Sites Implementing NAVIGATE Program (N = 17)

Site number	Number of clients enrolled RAISE-ETP	Location in country	Setting	Insurance Status	Public or not- for profit
1	18	Midwest	Urban, suburban, and rural	24% commercial 8% Medicaid 68% other	Not for profit
4	9	South	Suburban	0% commercial 30% Medicaid 70% other	Public
7	9	Midwest	Urban	48% commercial 39% Medicaid 13% uninsured	Not for profit
9	9	Northeast	Urban	20% commercial 50% Medicaid 30% other	Not for profit
11	14	Northeast	Urban and rural	5% commercial 95% Medicaid	Not for profit
14	6	South	Urban	5% commercial 45% Medicaid 50% other	Not for profit
15	14	West	Urban	50% commercial 50% other/uninsured	Not for profit
17	6	Midwest	Rural	20% commercial 80% Medicaid	Not for profit
18	19	Midwest	Urban	30% commercial 70% uninsured	Public
20	15	Midwest	Urban	5% commercial 63% Medicaid 32% others	Public
22	25	West	Suburban	2% commercial 18% Medicaid 80% uninsured	Not for profit
24	14	Northeast	Urban	6% commercial 43% Medicaid 51% other	Not for profit
26	4	Northeast	Suburban	5% commercial 45% Medicaid 50% unknown	Not for profit
28	11	South	Suburban	55% Medicaid 45% other/uninsured	Public
30	12	Midwest	Urban	3% commercial 67% Medicaid 29% other	Not for profit
33	16	West	Urban	20% Medicaid 80% other	Public
35	23	West	Urban	10% commercial 35% Medicaid 55% other	Public

2.2. Participants

For the purposes of the fidelity assessment discussed in this paper, the staff members at the 17 sites implementing the NAVIGATE program were the study participants.

2.3. NAVIGATE Treatment Program

NAVIGATE was designed to provide comprehensive care to individuals who have recently experienced an FEP (Mueser et al., 2015). The NAVIGATE program contains four integrated interventions, which were offered within a shared decision-making approach with clients selecting treatments based on their own treatment preferences and recovery goals. These interventions included: Individual Resiliency Training (IRT), the Family Education (FE) program, Supported Employment and Education (SEE), and Personalized Medication Management (PMM). These interventions have a clear evidence base (Bird et al., 2010; Killackey et al., 2008; Onwumere et al., 2011; Penn et al., 2005), and are considered key components of FEP services (Addington et al., 2013). The component services are described in more detail in Mueser et al. (2015) and Robinson et al. (2018).

2.3.1. Individual Resiliency Training (IRT).

IRT is an individual therapy program aimed at helping clients set and work towards personal goals, enhance wellness and personal resiliency, learn information about psychosis and treatment, and improve illness management(Meyer-Kalos et al., 2015). IRT is offered on a weekly or biweekly basis at enrollment in NAVIGATE and includes seven Standard modules recommended for all clients (e.g., Education about Psychosis), and six Individualized modules intended for delivery on an as-needed basis (e.g., Coping with Symptoms).

2.3.2. Family Education (FE) Program

The FE program was recommended for clients who were in regular face-to-face contact with family members (e.g., > 4 hours/week) or other significant people (e.g., girlfriend). The goals of the FE program are to teach families about psychosis and its treatment, to reduce relapses by monitoring of early warning signs, to provide support for the client’s work toward personal goals, and to reduce family stress (Mueser et al., 2015). Single family sessions are offered to all family members, including the client, who can choose to opt out of sessions if he or she prefers. A series of 10–12 sessions of psychoeducation is recommended for all families. Additional optional components of the FE program include family consultation to address circumscribed problems (1–2 sessions), and skills training to improve communication and problem solving skills. Brief monthly “check-ins” with the family clinician are encouraged after families have completed the program (Glynn et al., 2014).

2.3.3. Supported Employment and Education (SEE)

SEE was adapted from the Individual Placement and Support (IPS) model of supported employment (Becker and Drake, 2003) to include education, and is offered to all clients who want to work or resume their studies. SEE focuses on helping clients develop and pursue education and work goals, obtaining competitive jobs or enrolling in educational programs as rapidly as possible, and succeeding in work or school through provision of follow-along supports (Lynde et al., 2014). In order to ascertain whether clients have work or educational goals and want to receive SEE services, the SEE specialist endeavors to meet with each client at least once soon after their enrollment in the NAVIGATE program.

2.3.4. Personalized Medication Management (PMM)

PMM was recommended for all clients. NAVIGATE medication prescription includes detailed FEP psychotropic medication guidelines and a computerized decision support system named COMPASS to facilitate shared decision-making regarding prescriptions (Robinson et al., 2018). Antipsychotic medications are grouped based upon their documented efficacy and side effect profiles from the FEP and adolescent treatment trial literature into suggested treatment stages. Recommended dosing guidelines are provided for each medication. Suggested treatment begins with a stage 1 medication. If the stage 1 medication is ineffective, medications from subsequent stages (e.g., next a stage 2 medication, then if stage 2 was ineffective a stage 3 medication) are suggested. Guidelines for side effect minimization and for health monitoring and medical referral/treatment when applicable are also provided. As a part of COMPASS, prior to meeting with their prescriber participants record their recent symptoms and side effects on a standardized computer-based questionnaire, and their responses are summarized to provided to the prescriber for their meeting with the client.

2.3.5. The NAVIGATE Treatment Team

The four NAVIGATE interventions were provided by a multidisciplinary team that usually included five mental health professionals, who met regularly and worked together to work with clients towards achieving their personal goals (Mueser et al., 2014). The team was led by the director, who was usually a master’s level clinician and who also provided FE program, and who supervised the IRT clinicians and SEE specialist. The prescriber, a psychiatrist or nurse-practitioner, provided PMM. Two clinicians, usually master’s level, provided IRT, while one typically bachelor’s level member provided SEE. Case management was sometimes provided by one of these five members, or by a separate case manager who also served as a member of the team. Because of the relatively low flow of FEP clients at study sites, most team members were not employed full time on NAVIGATE, and served other clients at their agency. NAVIGATE clients could also access other services available at their local center.

2.4. Training of NAVIGATE Teams

NAVIGATE teams received a combination of in-person training and phone, as well as occasional video consultation that focused on both working effectively together as a team and implementing each of the specific NAVIGATE interventions. IRT and FE clinicians also received feedback from experts based on audio-files of sessions rated for adherence to the manuals using standardized fidelity scales as part of the certification process for training clinicians to implement the interventions. Following the initial inperson training, a series of training videos was created to demonstrate the implementation of IRT skills, which was made available to all IRT clinicians, and was used to train new clinicians.

2.4.1. In-person Trainings

At the initiation of the project, members of eight or nine NAVIGATE teams from different sites participated in three-day in-person training sessions conducted at a central location. Manuals for the overall program and the individual NAVIGATE interventions were distributed and reviewed at these meetings. The training was divided into team-based and individual specialty-based training. The team training was conducted with all of the team members together, and provided education about unique aspects of FEP, an overview of NAVIGATE and the specific roles of each member, and guidelines and exercises to foster effective teamwork (e.g., role playing a treatment team meeting). Individual specialty training was conducted concurrently for each of the four interventions, and included an introduction to the intervention, a review of the critical components, and a combination of modeling and role-playing skills for delivering it. In addition, directors received a half-day of training on leading the team, outreach to educate the community about the service and engage clients into NAVIGATE, strategies to maintain client engagement in treatment, and methods for supervising IRT and SEE.

Approximately two years after the initial training, an in-person two-day follow-up training meeting was conducted with all 17 teams. Half of a day was spent reinforcing team strengths and sharing success stories across the different teams. The remainder of time was devoted to concurrent advanced training in each intervention and the director role, which included reviewing common challenges, identifying solutions, demonstrating possible solutions, and engaging clinicians in role plays to practice specific strategies.

2.4.2. Expert Consultation

Ongoing expert phone consultation for each intervention and the director role was part of the implementation plan for NAVIGATE, and was described in the NAVIGATE Team Members’ Guide (Mueser et al., 2014). This consultation was an extension of the in-person training, and provided an opportunity for team members to get ongoing support and guidance as they implemented the NAVIGATE interventions. A specific consultation approach was developed for each intervention and the director, designed to provide the clinical training and support needed to achieve and maintain competence.

For IRT, as part of the training and fidelity evaluation, clinicians made audio-recordings of IRT sessions, which were uploaded to a secure website and listened to by an IMR consultant. IRT session fidelity was then rated by a consultant using a standardized form, and written quantitative and qualitative feedback was provided to the clinician and supervisor in a timely fashion (e.g., within a week of the session). This rapid turn-around time for providing feedback about the quality of sessions was designed to facilitate the shaping of clinicians’ skills early in the process of learning IRT, which has been used successfully to train frontline clinicians in implementing other psychosocial interventions for persons with severe mental illness (Lu et al., 2012).

Clinicians and the site IRT supervisor also received two 1-hour group phone consultations from an IRT expert monthly, with two sites participating per call, for the first four years of the project, with the frequency of consultations decreasing to monthly for the fifth year. During these calls, the group reviewed the status of clients engaged in IRT, consultants provided recommendations for implementing it, clinicians practiced skills and strategies, and any questions about written fidelity feedback from the consultant on audio-files of IRT sessions were addressed. There were additional opportunities during the calls to review and practice advanced IRT skills such as cognitive restructuring.

Clinicians providing the FE program also made audio-recordings of sessions, which were uploaded and listened to by a consultant who provided written quantitative and qualitative feedback in a timely fashion using a standardized fidelity scale. Consultation was offered weekly for 1-hour in a group format with 6–8 clinicians per call. FE clinicians were expected to join the calls at least twice a month for the first four years of the project, which was reduced to once per month for the fifth year. Clinicians could call in more frequently if needed to receive additional support or training. During these calls, the consultant reviewed with clinicians their active families in treatment, discussed strategies to engage new families in treatment, and provided additional training, support, and skills practice in FE as needed.

SEE experts provided group consultation on a biweekly basis to three to six SEE specialists per call. NAVIGATE directors were strongly encouraged to participate in these calls. The focus of the calls was on reviewing client engagement with the SEE specialist, client involvement in school and work activities, provision of SEE services (e.g., assessment of work/school interests and preferences, job development or liaising with schools), and problem-solving barriers to clients’ pursuit of work or school goals.

Prescribers received individual training via teleconferencing on technical aspects of the COMPASS decision support program. A monthly group teleconference with the NAVIGATE Central Team was also open to all prescribers, which included group feedback about clinical challenges and treatment options for these and review of relevant FEP literature. The COMPASS program provided prescribers real-time information on recommended NAVIGATE strategies for the treatment of symptoms and the management of medication side effects and medical health issues. Towards the end of the project, a call approximately every six months occurred between the Central Team and individual prescribers to provide an opportunity for case-by-case review.

Monthly group conference calls with three to four NAVIGATE directors per call were led by the consultants. Following identification of agenda items from directors on the call, these meetings followed a semi-structured agenda that included: review of recent NAVIGATE enrollees and dropouts, the number of IRT and SEE supervision meetings conducted, the number of team meetings conducted, and any challenges experienced implementing the components of the program. The remaining time was spent solving problems related to implementing NAVIGATE or addressing clinically challenging cases. The consultants also occasionally “sat in” (via phone) on different sites weekly NAVIGATE team meeting, and after provided feedback to the director (one to three times per site). Information gleaned from these informal consultations to the directors was not included in the fidelity assessments

2.4.3. Training New Team Members

New members of the NAVIGATE team were trained using a combination of strategies. Training for all new members included directed reading of manuals and related materials, individual or small group time with consultants to answer questions and ensure basic understanding of the intervention, and participation in ongoing consultation calls. For new IRT clinicians, IRT training tapes created early in the project were also employed in training new clinicians. Some new SEE specialists also took a 12-week online course on providing IPS supported employment as part of their training.

2.5. Fidelity Assessment

Specific methods for evaluating fidelity to each of the four NAVIGATE interventions were developed, as well as for evaluating the adherence of teams to the overall structure and staffing of the NAVIGATE program (Mueser et al., 2014). The methods used to measure fidelity were intervention specific, and depended on the types of information that could be readily accessed without imposing a significant burden on the sites. For IRT and FE, fidelity assessments were based on consultant reviews of audio-files of treatment sessions, using instruments that were adapted from fidelity measures of other psychosocial interventions for the severe mental illness population (Lu et al., 2012; McGuire et al., 2012). All IRT and FE sessions were audio-recorded, unless the client objected or there was equipment failure. Fidelity to SEE and the overall team were data. Copies of the IRT, FE, SEE, and NAVIGATE Team fidelity scales are included in the supplementary material for this article.

2.5.1. Individual Resiliency Training (IRT)

Fidelity to IRT was evaluated through a certification process, based on consultants’ ratings of audio-files of IRT sessions. Ratings were made on the IRT Fidelity Scale, a 14-item scale of critical components of IRT (e.g., agenda setting, goal setting/follow up, cognitive restructuring, skills training strategies), with each item rated on a 5-point Likert scale ranging from 1 (unsatisfactory) to 3 (satisfactory) to 5 (excellent) (Browne et al., 2016 Ahead of Print). Similar to some other scales for measuring fidelity to cognitive behavioral therapy (Muse and McManus, 2013), the IRT Fidelity Scale combines ratings of clinician adherence and competence, with low numbers for most items reflecting poor adherence to the treatment model (1, 2), and higher numbers reflecting level of competence for items that were adhered to (3–5). To ensure consistency of ratings, inter-rater reliability checks were conducted by having different consultants rate the same sessions, although these data were not analyzed.

Two levels of certification were established to designate clinicians who had demonstrated adequate fidelity to implementing the Standard (Level 1) IRT modules and the Individualized (Level 2) IRT modules. For both levels, clinicians were required to demonstrate an overall rating of at least 3 (satisfactory) on the IRT Fidelity Scale on four out of five consecutively rated sessions. A “3” was selected to indicate acceptable fidelity to the IRT model based on other cognitive behavioral therapy scales that employ a similar threshold to designate satisfactory clinician fidelity or competence (Blackburn et al., 2001; Haddock et al., 2001; Young and Beck, 1980). If fewer than four of the initial session ratings met the criterion, additional sessions were rated, and certification was met when four consecutive sessions met the criterion level. Level 1 certification was required before a clinician could obtain Level 2 certification. Following certification, quality ratings were conducted on approximately 10% of randomly selected sessions, with feedback sent to the clinician and supervisor.

2.5.2. Family Education (FE) Program

A similar certification process was used to evaluate fidelity to the FE program. The FE Fidelity Scale included 13 critical components of the program (e.g., agenda setting, use of family educational handouts and worksheets) that were rated on 5-point Likert scales. Similar to IRT, inter-rater reliability checks on ratings by different consultants were conducted, although data were not analyzed. Certification required clinicians to achieve a rating of 3 (satisfactory) or higher on the overall fidelity rating for three out of four sessions with two families. If the clinician did not meet this criterion for one or both families, sessions for additional families were rated using the same criteria, until the criteria were met for two families.

2.5.3. Supported Employment and Education (SEE)

The SEE Fidelity Scale (see Table 4) was developed to be completed based on a combination of program and administrative records (but not site visits), and included nine items scored by two raters on behaviorally anchored 4-point scales (1=poor, 2=limited, 3=basic, and 4=good) (Rosenheck et al., 2017). A score of “3” (“basic”) was considered the minimum for acceptable implementation.

Table 4:

Supported Employment and Education (SEE) Fidelity Scale Ratings for 17 Sites Providing the NAVIGATE Program

Site #
Fidelity Scale Item	1	4	7	9	11	14	15	17	18	20	22	24	26	28	30	33	35	Range	Overall Item Mean
1. SEE specialist caseload size	4	4	4	4	4	4	3	4	4	2	4	4	4	4	4	3	1	1–4	3.59
2. SEE specialist focuses only on work or school	4	4	4	4	4	4	4	4	4	4	2	4	4	2	4	4	2	2–4	3.65
3. Director and SEE specialist meet weekly for supervision	4	4	3	2	NA	2	4	4	2	NA	4	3	2	3	NA	4	2	2–4	3.07
4. SEE specialist attends weekly NAVIGATE team meetings	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4–4	4.00
5. SEE specialist attempts to engage all clients	3	4	4	4	4	4	3	4	4	3	4	4	4	3	3	3	3	3–4	3.59
6. Direct competitive job or integrated school search	3	4	4	4	4	4	4	4	4	3	2	2	4	2	3	3	2	2–4	3.29
7. Percent of SEE contacts in community	1	4	2	4	2	3	3	3	3	1	2	1	2	3	2	1	3	1–4	2.35
8. Follow-along supports offered or provided	2	4	2	3	4	3	4	4	3	2	3	3	4	3	3	3	1	1–4	3.00
9. SEE specialist and director attend consultation calls	3	3	3	2	3	3	3	4	2	1	3	4	2	3	2	2	3	1–4	2.71
SEE Fidelity Mean Quality Score by Site	3.11	3.89	3.33	3.44	3.63	3.44	3.56	3.89	3.33	2.50	3.11	3.22	3.33	3.00	3.13	3.00	2.33	-	3.25

Note: SEE fidelity rating descriptors: 4 = Good; 3 = Basic; 2 = Limited; 1 = Poor

Four items on the scale were based on four of the eight principles of IPS supported employment (zero-exclusion for eligibility: #5; focus on competitive work or integrated school: #6; integration of SEE and clinical treatment: #4; follow-along supports: #8), and five items were based on other characteristics of IPS included in the IPS Fidelity Scale (caseload size: #1; SEE specialist role: #2; supervision: #3 and #9; community-based services: #7) (Bond et al., 2012). Benefits counseling, a principle of IPS, was not included in the SEE Fidelity Scale because it was a responsibility of the entire NAVIGATE team and not just the SEE specialist (Mueser et al., 2014). SEE adaptations of the other three principles of IPS, including attention to client preferences, rapid job search, and job development, were not included in the scale because of difficulty rating them based on available records, although these principles was incorporated into the SEE Manual (Lynde et al., 2014).

2.5.4. Personalized Medication Management (PMM)

For the psychosocial interventions, sites were providing treatments that they had not provided before as these treatments had been developed or adapted from prior models specifically for NAVIGATE (e.g., sites could not have provided IRT before NAVIGATE as it did not exist before NAVIGATE was developed). In contrast, all sites had provided medication treatment to clients before NAVIGATE was developed and NAVIGATE medication recommendations only employed marketed agents available to all clients via prescription. Prescription for a particular client could be either 1) the site usual medication practice choice, or 2) a choice facilitated by the NAVIGATE guidelines (e.g., if a client received a prescription for risperidone, it might have been the prescription they would have received at the site outside of NAVIGATE treatment or it might have been from application of NAVIGATE guidelines).

To assess the degree to which a site followed NAVIGATE medication principles, we estimated the degree that sites’ prescriptions differed from usual practice. Medication prescriptions and dosage received by study participants were recorded monthly for both NAVIGATE and Community Care sites as part of study research procedures. Each month’s treatment was coded as either conforming or not to NAVIGATE first-line principles. A detailed description of these principles has been published (Robinson et al., 2018). The site-specific metrics were averaged to determine the median percent months of first-line treatment across all sites (NAVIGATE and Community Care). Site-specific rates were compared with the median all-sites rate to determine if first-line prescription rates at a site were greater or lesser than the overall median rate.

2.5.5. NAVIGATE Team Composition and Activities

Adherence of each team to the structural and process elements related to team activities was evaluated by two raters based on a combination of program and consultant records, with the 10-item behaviorally anchored NAVIGATE Team Fidelity Scale, which employs 4-point anchored scales with the same descriptors as the SEE Fidelity Scale. Team fidelity was assessed regarding the defining characteristics of the program described in the NAVIGATE Team Members’ Guide, including continuity of staffing and services, participation of all staff at weekly team meetings, and director supervision of IRT and SEE.

2.5.6. NAVIGATE Fidelity Index

The NAVIGATE Fidelity Index was developed in order combine the five component fidelity ratings into an overall measure of a team’s adherence to the NAVIGATE model. Three-point scales were created to summarize the different fidelity components with respect to the adequacy of their implementation: 1 = not implemented, 2 = basic implementation, 3 = good implementation. A mean of these five scores was computed to form an overall NAVIGATE Fidelity Index score for each site ranging from 1 to 3. NAVIGATE programs with Index scores < 2 were designated “not implemented,” scores ≥ 2 and < 2.5 were designated “basic implementation,” and scores ≥ 2.5 were designated “good implementation.”

For IRT, sites that had no clinicians certified in IRT Level 1 were given a Fidelity Index score of 1, sites with at least 1 clinician certified in IRT Level 1 but no clinicians certified in IRT Level 2 were given an Index score of 2, and sites with at least 1 clinician certified in IRT Level 2 were given an Index score of 3. For FE, sites that had no clinicians certified in FE were given an Index score of 1, and sites that had at least 1 clinician certified in FE were given an Index score of 3.

For both SEE and NAVIGATE team fidelity, the mean scores on each fidelity measure were used to assign Index scores as follows: 1 = mean score < 3 (“basic”), 2 = mean score 3–3.5, 3 = mean score > 3.5. For fidelity to PMM, sites in which the mean percentage adherence to the antipsychotic guidelines was above the median percentage adherence for all 34 study sites were given an Index score of 3, and sites with mean adherence below the median were given an Index score of 1.

2.6. Receipt of NAVIGATE Program Services

For PMM, the COMPASS computer system recorded data from every visit. To evaluate participation in the three NAVIGATE psychosocial interventions, we examined client responses to selected items on the Service Use Reporting Form (SURF) (Rosenheck and Fontana, 2003). The SURF is a brief instrument that was administered to all study participants (including those at Community Care sites) on a monthly basis in order to obtain information about recent service utilization, and which included three questions designed to evaluate whether participants had received IRT, FE, and SEE, and if so how many sessions or meetings of each (Kane et al., 2016).

2.7. Statistical Analyses

We summarized the demographic (gender) and professional characteristics of the staff members on the NAVIGATE teams across the 17 sites, as well as the time spent on the study, by computing means or percentages for staff based on their primary role on the team. In order to evaluate whether staff members fulfilling different roles on the team (director, prescriber, IRT specialist, SEE specialist) differed significantly in the length of time they participated in the study, a one-way analysis of variance was performed.

In order to evaluate the participation rate in PMM, we computed the percentage of clients who completed at least one visit based on the COMPASS computer system, and among those the mean number of visits. For participation in IRT, FE, and SEE we computed the percentage of clients who reported receiving at least one service for each intervention, and among those the mean number of services received over the two-year study period.

For IRT, we computed the number of sites that had at least one clinician who was certified in IRT Level I, and the number of sites with a clinician certified at IRT Level 2. Similarly, for FE we computed the number of sites with at least one clinician certified in FE. For IRT certification at both Levels 1 and 2, and FE certification, we also calculated the mean number of sessions rated and the duration of time required for clinicians to achieve certification. For the SEE Fidelity Scale and NAVIGATE Team Fidelity Scale we computed the mean score and range for each item, and the mean rating across items for each site.

In order to evaluate whether fidelity to the different components of NAVIGATE across sites were correlated with each other, Pearson correlations were computed between the three-point fidelity scores that comprised the NAVIGATE Fidelity Index.

3. Results

A total of 129 practitioners served on NAVIGATE teams at the 17 sites participating in RAISE-ETP study. Table 2 summarizes the characteristics of these practitioners, organized according to their role on the team. Most of the SEE specialists had a bachelor’s degree, whereas the majority of IRT providers, project directors, and family clinicians had master’s degrees. Among the prescribers, 80.0% were medical doctors (MDs). The mean number of months on the project ranged from 30.4 months for SEE specialists to 38.8 months for directors, and did not differ significantly between the NAVIGATE staff roles, F(4,116)=0.64, NS.

Table 2:

Characteristics of Treatment Providers of NAVIGATE Program (N = 129)

Demographics	Primary Role on NAVIGATE Team
	Director/Family n=15	Director n=11	Prescriber n=30	Family n=7	IRT n=44	SEE n=22
Gender Frequency Female (%)	9 (60.0)	8 (72.7)	11 (36.7)	5 (71.4)	32 (72.7)	16 (72.7)
Months on Project Mean (SD)	32.9 (16.3)	38.8 (13.2)	36.4 (15.7)	34.2 (13.5)	33.6 (13.7)	30.4 (16.5)
Years in Field Mean (SD)	14.5 (9.9)	13.7 (6.1)	21.4 (8.5)	19.4 (7.1)	12.6 (10.4)	7.3 (6.8)
Highest Level of Education Frequency (%)
High School	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	1 (4.6)
Associate’s Degree	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	3 (13.6)
Bachelor’s Degree	0 (0.0)	2 (18.2)	0 (0.0)	0 (0.0)	4 (9.1)	12 (54.6)
Master’s Degree	13 (80.0)	6 (54.6)	6 (20.0)	3 (42.9)	33 (75.0)	2 (9.1)
Doctoral Degree	2 (20.0)	3 (27.3)	24 (80.0)	2 (28.6)	6 (13.6)	0 (0.0)
Not Available	0 (0.0)	0 (0.0)	0 (0.0)	2 (28.6)	1 (2.3)	4 (18.2)
Highest Licensure Attained Frequency (%)
No License	0 (0.0)	0 (0.0)	0 (0.0)	2 (28.6)	0 (0.0)	15 (68.2)
LPCC/LMFT/LCSW	10 (66.7)	4 (36.4)	0 (0.0)	1 (14.3)	26 (59.1)	0 (0.0)
Psychologist	2 (13.3)	1 (9.1)	0 (0.0)	1 (14.3)	4 (9.1)	0 (0.0)
RN/LPN	0 (0.0)	1 (9.1)	0 (0.0)	1 (14.3)	0 (0.0)	1 (4.6)
NP	0 (0.0)	0 (0.0)	6 (20.0)	0 (0.0)	0 (0.0)	0 (0.0)
MD	0 (0.0)	0 (0.0)	24 (80.0)	0 (0.0)	0 (0.0)	0 (0.0)
Not Available	3 (20.0)	5 (45.5)	0(0.0)	2 (28.6)	14 (31.8)	6 (27.3)

Notes: LPCC = Licensed Professional Clinical Counselor, LMFT = Licensed Marriage and Family Therapist, LCSW = Licensed Social Worker, RN = Registered Nurse, LPN = Licensed Practical Nurse, NP = Nurse Practitioner, MD = Medical Doctor

Of the 223 study participants at NAVIGATE sites, according to the COMPASS program 211 (94.6%) completed one or more PMM visits, and among those they had a mean of 14.2 visits. Based on the monthly SURF reports, 205 (91.9%) participants reported receiving at least one IRT service (M=24.1), 150 (71.3%) reported receiving an FE service (M=13.7), and 187 (83.9%) reported an SEE service (M=13.6).

3.1. Individual Resiliency Training (IRT) and Family Education (FE) Interventions

The characteristics of certification for clinicians providing IRT and FE are summarized in Table 4. For IRT, 36 of 42 clinicians (85.7%) achieved Level 1 (Standard Modules) certification; at least one clinician was certified at each of the 17 sites. Fourteen clinicians (32%) achieved Level 2 (Individualized Modules) certification, including at least one clinician at 11 sites (65%). For FE, a total of 19 out of 22 clinicians (86%) achieved certification; 15 of the 17 sites (88%) had at least one certified clinician. For both IRT Levels 1 and 2, the number of sessions required to achieve the four acceptable sessions required for certification was 5.17 and 4.29 sessions, respectively. Clinicians required a mean of 9.63 sessions to achieve the eight acceptable sessions for FE certification. Thus, most of the sessions recorded by clinicians for certification in both IRT and FE were rated as satisfactory or higher quality.

The results of the certification process leave open the question of how much FE was provided by clinicians who were not certified, and how much IRT was delivered by clinicians who were not certified at one or both levels of IRT. Among the 3 clinicians who were not certified in FE, a total of only 14 sessions were audio-recorded, suggesting that the vast majority of FE sessions were provided by certified clinicians. Similarly, among the 6 clinicians who were not certified in the Standard Modules of IRT, a total of only 13 sessions were audio-recorded, also suggesting that most Standard Module IRT sessions were provided by clinicians certified at that level.

While relatively few Standard Module IRT sessions were provided by clinicians who were not certified at that level of IRT, more Individualized Module IRT sessions (N=48) were provided by the 22 clinicians who never achieved that higher level of certification. We compared the fidelity of the Individualized Module IRT sessions between clinicians who were certified at that level and clinicians who were not certified at the same level by conducting a t-test on the overall session quality rating of the IRT Fidelity Scale. The t-test was significant, t=2.75, df=105, p=.007, with certified clinicians having higher quality ratings (M=3.63, SD=0.72) than non-certified clinicians (M=3.23, SD=0.78). However, the average quality rating of the non-certified clinicians was nevertheless above the “satisfactory” rating of 3 on the IRT Fidelity Scale. Thus, while only a minority of sites had clinicians who were certified in the Individualized IRT Modules, this finding suggests that acceptable levels of quality were achieved even when non-certified clinicians delivered these modules.

3.2. Supported Employment and Education (SEE)

Data for the SEE Fidelity Scale ratings for the sites are presented in Table 4. Four sites (24%) were in the upper range of basic to good fidelity (≥ 3.5), 11 (65%) were in the lower range of basic to good fidelity (≥ 3.0 and < 3.5), two sites (12%) were in the range of limited to basic fidelity (≥ 2.0 and < 3.0). Considering a mean SEE score corresponding to “basic” fidelity as the minimum acceptable fidelity, 15 of the 17 sites (88.2%) implemented SEE with acceptable levels of fidelity.

3.3. Personalized Medication Management (PMM)

As presented in Table 5, the median percentage of months across all sites that participants received first-line prescriptions over the potential 2-year follow-up was 41.37%. Twelve of the 17 NAVIGATE sites (70.5%) had a site-specific percentage of greater than the median and 5 were below the median. In contrast, only 5 of the 17 Community Care sites had a site-specific percentage above the median and 12 were below the median, a statistically significant difference (χ² = 5.76, N=17 p=.016)_.

Table 5.

Percent of Time by Site that Patients were Prescribed Medications Conforming to NAVIGATE Personalized Medication Management (PMM) First-line Treatment Guidelines¹

Site number	Site Condition	Percent first- line antipsychotic treatment	Did the NAVIGATE site have a percentage greater than the median for all sites?	Did the Community Care site have a percentage greater than the median for all sites?
1	NAVIGATE	70.78%	yes
2	Community Care	70.33%		yes
3	Community Care	36.15%		no
4	NAVIGATE	45.45%	yes
5	Community Care	16.60%		no
6	Community Care	39.49%		no
7	NAVIGATE	43.55%	yes
8	Community Care	74.03%		yes
9	NAVIGATE	39.84%	no
10	Community Care	40.79%		no
11	NAVIGATE	33.38%	no
12	Community Care	38.57%		no
13	Community Care	37.55%		no
14	NAVIGATE	25.22%	no
15	NAVIGATE	59.55%	yes
16	Community Care	25.07%		no
17	NAVIGATE	37.83%	no
18	NAVIGATE	48.48%	yes
20	NAVIGATE	47.11%	yes
21	Community Care	43.00%		yes
22	NAVIGATE	51.34%	yes
23	Community Care	48.88%		yes
24	NAVIGATE	62.07%	yes
25	Community Care	41.96%		yes
26	NAVIGATE	89.29%	yes
27	Community Care	20.37%		no
28	NAVIGATE	58.72%	yes
29	Community Care	30.94%		no
30	NAVIGATE	78.77%	yes
32	Community Care	39.65%		no
33	NAVIGATE	28.47%	no
34	Community Care	38.29%		no
35	NAVIGATE	55.49%	yes
36	Community Care	10.00%		no
	Median percent months with first line antipsychotic treatment based upon data from all sites	41.37%
	Number of sites above median for months of first line antipsychotic treatment		12	5

¹For each site, the percent of time that each participant received a prescription for medications conforming to NAVIGATE first-line treatment guidelines was calculated and these values were averaged to calculate the site metric

3.4. NAVIGATE Team Fidelity

Data for the NAVIGATE Team Fidelity Scale are provided in Table 6. Eleven sites (65%) were in the upper range of basic to good fidelity (≥3.5), 5 sites (29%) were in the lower range of basic to good fidelity (≥3.0 and <3.5), and 1 (6%) was in the upper range of limited to basic fidelity (≥2.5 and <3.0). If “basic” fidelity (M≥3) is considered the minimal acceptable level of adherence to the structure and staffing of NAVIGATE model, then only one of the 17 sites (6%) fell below an acceptable level.

Table 6:

Team Fidelity Scale Ratings for 17 Sites Providing the NAVIGATE Program

Site #
Fidelity Scale Item	1	4	7	9	11	14	15	17	18	20	22	24	26	28	30	33	35	Range	Overall Item Mean
1. Continuity of staffing	3	3	4	4	4	4	4	4	2	4	4	4	4	4	4	4	3	2–4	3.71
2. Continuity director services	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4–4	4.00
3. Continuity IRT services	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4–4	4.00
4. Continuity FE services	4	1	4	4	4	4	4	4	4	4	4	4	4	4	4	4	2	1–5	3.71
5. Continuity SEE services	1	4	4	4	4	4	4	4	4	3	4	4	4	2	4	4	4	1–5	3.65
6. Continuity PMM services	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4	4–4	4.00
7. Weekly team meetings	4	4	4	2	4	4	4	4	4	2	4	3	3	3	3	4	3	2–4	3.47
8. All staff at team meetings	3	3	4	3	1	1	3	2	4	3	4	2	2	1	4	3	2	1–4	2.65
9. Director supervises IRT	4	3	4	2	4	3	4	3	2	1	3	3	3	2	1	4	1	1–4	2.76
10. Director supervises SEE	4	4	4	3	3.5	3	4	4	3	3.5	4	4	2	4	3.5	4	2	2–4	3.50
Team Mean Fidelity Score by Site	3.50	3.40	4.00	3.40	3.65	3.50	3.90	3.70	3.50	3.25	3.90	3.60	3.40	3.20	3.55	3.90	2.90	-	3.54

Notes: Team fidelity descriptors: 4 = Good, 3 = Basic, 2 = Limited, 1 = Poor; FE = Family Education program; IRT = Individual Resiliency Training; PMM = Personalized Medication Management; SEE = Supported Employment and Education

3.5. NAVIGATE Fidelity Index

The mean adherence of sites to the NAVIGATE model (i.e., the NAVIGATE Fidelity Index), including fidelity to each of the four treatments and the NAVIGATE staffing and structure is provided in Table 7. Nine sites (53%) were in the “good implementation” range (M>2.5 – 3.0), eight (47%) were in the “basic implementation” range (M=2.0–2.5), and no sites were in the “not implemented” range (M<2.0).

Table 7:

NAVIGATE Fidelity Index for 17 Sites Implementing the NAVIGATE Program

	Overall Fidelity Scores
Site #	Team	SEE	IRT	FE	PMM	Mean Score/ Site
1	3	2	3	3	3	2.80
4	2	3	2	1	3	2.20
7	3	2	3	3	3	2.80
9	2	2	3	3	1	2.20
11	3	3	3	3	1	2.60
14	3	2	3	3	1	2.40
15	3	3	3	3	3	3.00
17	3	3	3	3	1	2.60
18	3	2	3	3	3	2.80
20	2	1	2	3	3	2.20
22	3	2	3	3	3	2.80
24	3	2	3	3	3	2.80
26	2	2	2	1	3	2.00
28	2	2	2	3	3	2.40
30	3	2	2	3	3	2.60
33	3	2	2	3	1	2.20
35	1	1	3	3	3	2.20
Total Score	44	36	45	47	41	-
Overall Mean Score	2.59	2.12	2.65	2.76	2.41	2.51

Notes: FE = Family Education program; IRT = Individual Resiliency Training; PMM = Personalized Medication Management; SEE = Supported Employment and Education; Rules for scoring individual components of NAVIGATE Fidelity Index (NFI):

NAVIGATE Team Fidelity Scale: NFI = 3 if mean score 4 – 3.5; NFI = 2 if score 3.5 – 3.0; NFI = 1 if score < 3.0

Supported Employment and Education (SEE) Fidelity Scale: NFI = 3 if mean score 4 – 3.5; NFI = 2 if score 3.5 – 3.0,; NFI = 1 if score < 3.0

Individual Resiliency Training (IRT): NFI = 3 if ≥ 1 person at site is certified in IRT standard and individualized modules; NFI = 2 if ≥ 1 person at site is certified in IRT standard modules; NFI = 1 if 0 persons at site are certified in IRT standard modules

Family Education (FE): NFI = 3 if ≥1 person at site is certified in FE; NFI =1 if 0 persons at site are certified in FE

Personalized Medication Management (PMM): NFI = 3 if mean adherence is ≥ 67%; NFI = 2 if adherence is 33–66%; NFI = 1 if adherence is ≤ 33%

Two Pearson correlations between the five different fidelity ratings for each site included in the NAVIGATE Fidelity Index were significant: fidelity to IRT was correlated with fidelity to FE (r=.49, p=.04), and NAVIGATE Team fidelity was correlated with fidelity to SEE (r =.49, p=.04). None of the other correlations were significant.

4. Discussion

The findings indicated that among the 17 NAVIGATE sites in the RAISE-ETP project, all demonstrated at least basic or higher levels of fidelity to the model, according to scores on the NAVIGATE Fidelity Index. Fidelity to NAVIGATE was measured using clear definitions for each of the four interventions included in the program, as well as the structure and staffing of the program. This is an important finding because one of the requirements stated by the NIMH Request for Proposals for the RAISE initiative was that the intervention could be delivered in real-world settings (Kane et al., 2015). More than 130 practitioners provided NAVIGATE treatment to clients and on average, they were part of a NAVIGATE team for more than three of the five years that the study was ongoing at their sites. This means that many participants saw continuity in their treatment providers.

Among the five individual components of the NAVIGATE Fidelity Index, sites scored the highest on adherence to the NAVIGATE team structure and staffing, and the FE and IRT treatments; mean Index scores for these items were over 2.5. Sites were generally effective at hiring and replacing staff on the NAVIGATE team, offering the range of services in the model, and meeting regularly for team meetings and supervision.

Further, all 17 sites had at least one clinician certified in the IRT Standard modules, 11 sites (65%) had a clinician certified in the IRT Individualized modules, and 15 sites (88%) had a clinician certified in the FE program, indicating high rates of fidelity to these psychotherapeutic components of the NAVIGATE program. In addition, the number of audio-files of IRT and FE sessions reviewed by fidelity raters for certification was only 5.15 sessions to achieve 4 acceptable sessions for the IRT Standard modules, 4.29 sessions to achieve 4 acceptable sessions of IRT Individualized modules, and 9.63 sessions to achieve 8 acceptable sessions of FE. These findings suggest that the combination of the training, manuals, supervision, and consultation for IRT and FE were sufficient for clinicians to rapidly demonstrate good clinical skills when providing each intervention for the first time. Anecdotally, the limited amount of time to implement NAVIGATE (i.e., a maximum of two years of study enrollment, and often less time), and the slow enrollment of clients at some sites, led to low caseloads of FEP clients and engagement of family members in treatment, making it more difficult for clinicians to achieve certification in the FE program and IRT Individualized modules.

The model for assessment of PPM fidelity differed from the certification procedures for IRT and FE. Prescribers were either psychiatrists or nurse-practitioners and therefore licensed to prescribe marketed antipsychotics, which were the only ones used the study. As a result, the fidelity model here was to contrast the prescribing practices at NAVIGATE sites to an estimate of usual practice. NAVIGATE prescribers were significantly more likely to prescribe antipsychotics that were in the first tier of recommendations according to the NAVIGATE guidelines. These results suggest that support provided by written guidelines, the web-based COMPASS decision support system, and the training model had a valuable impact. However, it is important to note that for PMM a decision support system such as COMPASS is not readily available in routine practice.

In contrast to the relatively strong implementation of IRT, FE, PPM, and the overall NAVIGATE team structure, the implementation of SEE was somewhat weaker, with the mean NAVIGATE Fidelity Index score for this item of 2.12, just above the “basic” implementation level. Some of the challenges in implementing the SEE program have been previously discussed (Rosenheck et al., 2017), including the lack of financing mechanisms for SEE at some sites. Despite supplementary research funds available to support SEE, some sites could not adequately support SEE services, making it challenging to implement with high fidelity to the model. As with supported employment (Drake et al., 2016; Mueser and Cook, 2016), more reliable funding mechanisms are needed to support the provision of SEE to the FEP population. Other factors may have also contributed to attenuated SEE fidelity, as discussed below.

Among the correlations between elements of the NAVIGATE Fidelity Index, two were statistically significant: fidelity to the IRT and FE programs (r = .49), and fidelity to SEE and the overall NAVIGATE team (r = .49). These correlations could reflect shared method variance between how fidelity to the elements of NAVIGATE were measured. Fidelity to IRT and FE were evaluated with a certification process based on audio-files of sessions, whereas fidelity to SEE and the NAVIGATE team were evaluated by review of administratively collected data on services provided (SEE) and team staffing and activities. However, the ratings of NAVIGATE program elements were made by experts in each area and not by the same raters, somewhat attenuating this possibility.

The correlation between IRT and FE fidelity could also reflect the effects of site-related factors on the implementation of these two psychotherapeutic interventions, such as caseload size, cross-training of clinicians in both interventions, the ability of stronger sites to hire better clinicians, and turnover of the director position, who usually provided FE and supervised the IRT clinicians. In addition, the significant association between fidelity to the SEE model and fidelity to the NAVIGATE team model points to the potential influence of systemic factors in successful implementation (Aarons et al., 2011). Difficulties implementing the SEE program due to limited funding for these services, and problems maintaining the continuity of staffing, services, and responsibilities of different team members, could reflect broader issues related to the resources available at sites to support the implementation of NAVIGATE, and the capability and will of leadership in commanding those resources. Access to resources and quality of leadership are frequently cited factors in contributing to the success of implementing novel psychosocial interventions in community settings (Lundgren et al., 2013; McGuire et al., 2015; Whitley et al., 2009).

The method for evaluating fidelity to the NAVIGATE program differed in important ways from some other methods used for CSC programs (Radhakrishnan et al., 2017 Ahead of Print), including the First Episode Psychosis Services Fidelity Scale (FEPS-FS) (Addington et al., 2016). The FEP-FS was developed in order to identify the critical components of a range of empirically supported programs for persons with FEP through a systematic review of the research literature followed by a Delphi consensus process (Addington et al., 2013). The resulting scale was designed to be completed by two or three assessors based on a site visit, and to extract information through a combination of interviews, record reviews, and observations (Addington et al., 2016). In contrast, the assessment of fidelity to the NAVIGATE program was intended to evaluate adherence and competency to a specific set of interventions and defined program structure, standardized in a set of manuals. Thus, a more precise approach to measuring fidelity to this program was possible based primarily on evaluation of individual providers of the interventions.

Assessments based on reviews of audio-files of IRT and FE sessions were the most time consuming methods used to evaluate fidelity. However, the review of these sessions also provided quantitative and qualitative feedback to clinicians, which was an integral part of their training.

Therefore, the time required to implement this component of the fidelity assessment should not viewed in isolation as a program monitoring cost, but instead should be considered within the broader context as a cost related to the high-quality training and supervision of clinicians.

Several limitations of this study should be noted. First, PPM included an extensive range of recommendations, including strategies for adherence enhancement, treatment for clients with varying degrees of treatment resistance, side effect minimization and general medical management.

To include all recommendations in a fidelity measure would have resulted in a measure that would have been so complex that it would be difficult to interpret. Instead, for fidelity assessment we focused upon one key recommendation, prescription of a NAVIGATE first-line antipsychotic.

Second, the 17 sites providing NAVIGATE were not necessarily nationally representative of mental health centers in the U.S. Rather, sites participating in the study were chosen following an open, national solicitation process in which potentially eligible and interested mental health centers applied to participate in the study (Kane et al., 2015). It is likely that participating centers were more open to innovation and interested in learning new service models than the average mental health center, which could have facilitated the implementation of NAVIGATE. While these “early adopters” may have been motivated to learn this new treatment model (Panzano and Roth, 2006), academic sites or those that already had an FEP program were excluded from participation. Thus, aside from having a sufficient number of clients and staff members to participate in the study, and potentially greater enthusiasm for innovative programs among the agency leadership, these sites had no special advantages over other community-based mental health agencies serving people with FEP.

These limitations notwithstanding, the present results demonstrate that the NAVIGATE program can be implemented with acceptable levels of fidelity with existing staff at typical community mental health care centers. Considering that the primary findings from the cluster randomized controlled trial showed that over two years participants at NAVIGATE sites had substantially better clinical and psychosocial outcomes than Community Care sites, priority should be given to disseminating this program throughout the U.S. Research is needed to develop more efficient methods of implementing the NAVIGATE program, and training team members on the key interventions included within the program.

Table 3:

Summary of Fidelity Certification of Clinicians Providing Individual Resiliency Training (IRT) and Family Education (FE) Interventions in NAVIGATE Program

	IRT Fidelity		FE Fidelity N(%)
	Standard N(%)	Individualized N(%)
Total Clinicians Certified	36 (82%)	14 (32%)	19 (86%)
Sites with 1 Certified Clinician	17 (100%)	11 (65%)	15 (88%)
	M (SD)	M (SD)	M (SD)
Mean sessions to Certification	5.17 (1.67)	4.29 (0.61)	9.63 (2.54)
Mean length of time (months) to Certification	5.23 (4.55)	11.78 (6.35)	7.13 (5.98)
Mean fidelity scale score	3.25 (0.74)	3.50 (0.93)	3.11 (0.72)

Notes: IRT Certification for Standard and Individualized Modules include 4 consecutive sessions rated on overall item of satisfactory (3) or above.

FE certification included ratings of satisfactory (3) or above on 4 sessions for 2 families (total 8 sessions).