The First Episode Psychosis Services Fidelity Scale 1.0: Review and Update

Abstract

The First Episode Psychosis Fidelity Scale, first published in 2016, is based on a list of essential components identified by systematic reviews and an international consensus process. The purpose of this paper was to present the FEPS-FS 1.0 version of the scale, review the results of studies that have examined the scale and provide an up-to-date review of evidence for each component and its rating. The First Episode Psychosis Services Fidelity Scale 1.0 has 35 components, which rate access and quality of health care delivered by early psychosis teams. Twenty-five components rate service components, and 15 components rate team functioning. Each component is rated on a 1–5 scale, and a rating of 4 is satisfactory. The service components describe services received by patients rather than staff activity. The fidelity rater completes ratings based on administrative data, health record review, and interviews. Fidelity raters from two multicenter studies provided feedback on the clarity and precision of component definitions and ratings. When administered by trained raters, the scale demonstrated good to excellent interrater reliability. The selection of components can be adjusted to rate programs serving patients with bipolar disorder or an attenuated psychosis syndrome. The scale can be used to assess and improve the quality of individual programs, compare programs and program networks. Researchers can use the scale as an outcome measure for implementation studies and as a process measure for outcome studies. Future research should focus on demonstrating predictive validity.

Introduction

Fidelity scales are an important tool for successful implementation of evidence-based practices, and attention to their psychometric properties is important.¹ Integrated team-based care for first episode psychosis is now an evidence-based practice. There is extensive evidence supporting the superiority of intensive, team-based care for treating a first episode of psychotic disorders.² A large cluster randomized controlled trial in the United States demonstrated the cost-effectiveness of team-based intensive first episode psychosis services.³ Implementation of first episode psychosis services varies internationally.⁴ Where implementation is most consistent, fidelity scales or quality indicator sets have been used to assess the degree of implementation.⁵ The challenge in designing a fidelity scale is to develop a practical method that is rigorous, but feasible to apply to all programs. Fidelity scales need to be convincing to funders, service delivery organizations, clinicians, and service users.⁶

Overview of the First Episode Psychosis Services Fidelity Scale (FEPS-FS 1.0)

The First Episode Psychosis Services Fidelity Scale (FEPS-FS) was developed using standardized methodology for developing fidelity scales.⁷ The original version first published in 2016⁸ was based on a list of essential components identified by systematic reviews and an international consensus process.⁹ The scale and manual have been revised based on feedback from clinicians, researchers, and funders during the course of two multicenter studies.^10,11 Fidelity raters from two multicenter studies provided feedback on the clarity and precision of component definitions and ratings. The revised scale and manual showed good-to-excellent inter-rater reliability when used by trained raters.¹⁰ The current 1.0 version of the scale has 35 components which rate access and quality of health care delivered by early psychosis teams.¹² Each component is rated on a 1–5 scale, in which a rating of 4 is satisfactory. The service components describe services received by patients rather than staff activity. The fidelity rater completes ratings based on administrative data, health record review, and interviews with clinicians. The selection of components can be adjusted to rate programs serving patients with bipolar disorder or those at clinical high risk for psychosis.

The purpose of the current paper is to present the FEPS-FS 1.0 version of the scale, review the results of studies which have examined the scale, and provide an up-to-date review of evidence for each component and its rating.

Methods

This paper uses a “literature review,” a type of review which uses published materials that provide examination of recent or current literature. It can cover a wide range of subjects at various levels of completeness and comprehensiveness.¹³ The review included studies using the FEPS-FS which used both quantitative and qualitative analyses. In addition, it covered literature supporting each FEPS-FS component, and the ratings for each component. The levels of evidence ranged from high, when supported by a recent systematic review or meta-analyses, to the level of good clinical practice for items addressing clinical assessment.

Results

Review of Studies with the FEPS-FS

The scale was first tested in six programs in Canada and the United States and demonstrated both feasibility and reliability. Fidelity ratings were conducted by expert raters during onsite visits.⁸ The scale has since been shown to be feasible and acceptable when used by trained clinician raters working with healthcare evaluators during site visits.¹⁴ In this cross-sectional study of 9 programs, the scale was evaluated based on assessor focus groups, program staff interviews, data from raters’ consensus meetings, and time-tracking logs. A general inductive approach for analyzing qualitative data was used, and quantitative data were aggregated and summarized. Fidelity rater feedback was positive and indicated that use of peer assessors and the in-person site visit added value to the process. It was generally perceived that the model provided valuable information to assist internal quality improvement efforts. Further, assessors reported direct benefits from participating, including networking and learning opportunities. The fidelity raters provided important feedback on clarity of component descriptions and the structured interview, resulting in interview revisions and clarification of rating criteria in the manual. The fidelity ratings from the study demonstrated that the FEPS-FS captured variation in program implementation and provided a baseline for measuring change.¹¹

Investigators in Italy used an Italian translation of the scale as a self-report measure to assess the quality of services in 29 programs. The results demonstrated variability in the quality of care delivered by individual programs and some deficiencies common to programs.⁵

The Mental Health Block Grant 10% Set Aside study of 36 programs receiving Federal Government funding in the United States provided the opportunity to first revise the scale for remote assessment and then test the revisions [http://nri-inc.org/media/1620/2-state-involvement-in-csc-programs.pdf]. Fidelity was assessed at 2 time points 1 year apart using a remote fidelity assessment process. De-identified administrative data provided by the programs were uploaded to a secure website. Local staff were trained in data abstraction from selected health records using a standardized template uploaded to the secure website. Finally, a trained fidelity rater completed structured telephone interviews with staff. During the first year, the scale was revised based on feedback from the fidelity raters.

The modifications to the scale based on the feedback from both the Canadian and the first year of the US study resulted in a revised version of the scale. The scale was made more concrete with less scope for interpretation. Some components were dropped because they proved hard to measure reliably. The rating criteria were made more consistent. The interview was made into a structured interview and more clearly linked to ratings. In the second year, the revised version of the scale was tested for inter-rater reliability. Based on 5 programs and 4 raters, inter-rater reliability was in the good-to-excellent range, with a mean ICC of 0.91 (95% confidence interval = 0.72–0.99, P = 0.001).¹⁵ Feasibility was further supported by the remote assessment study, which indicated that data collection during the second year required only 10.5 h of program time, and the interviews were completed in 5 h.

In 2020, the fidelity scale was modified by adding two components, one on assessment of fidelity, a component recommended in the Coordinated Specialty Care model.¹⁶ The second on the age range served as recommended by the National Institute of Health and Care Excellence.¹⁷ The addition of these two components should not change the reliability of the scale because both the age range served by the program and whether or not they use a fidelity scale are clear and easily measured components. The reliability of a scale lies both on the structure of the scale and on the training of the raters; as a result, reliability should be assured within research studies.¹⁸ The changes in the scale can be seen by comparing the original published version with the current version.⁸

General Changes to the Scale and Manual

Two general challenges were identified when using the scale. One was how to deal with diagnostic heterogeneity because some programs served a range of diagnostic groups. The second was to support the reliability of the self-report fidelity process.

A systematic review and meta-analysis of the cost-effectiveness of early intervention services identified variability in the diagnostic groups served by early psychosis programs.¹⁹ Individual programs serve patients with schizophrenia spectrum disorders, clinical high risk for psychosis bipolar disorder, and mood disorders with psychotic features. To address this challenge, the fidelity scale includes a component that requires the program to identify the diagnostic categories served by the program (see table 1, Component 8). The fidelity process which is described in the manual then requires the program to identify the numbers in each group. Next in a process described in the manual, the selection of components used to calculate fidelity is modified for each diagnostic group and a separate analysis of fidelity conducted for groups that include sufficient numbers. For example, 3 pharmacotherapy items (table 1) including items 19 (antipsychotic prescription), 20 (antipsychotic dosing), and 21 (clozapine use) are only applied to patients with a schizophrenia spectrum disorder and not to patients at clinical high risk for psychosis.

Table 1.

Research Supporting Individual Components and Ratings

Component	Supporting Evidence
1. Practicing Team Leader	The role of the practicing team leader is reportedly a key ingredient for the successful implementation of evidence-based practice in adult mental health. Important behaviors include facilitating team meetings, building and enhancing staff skills, monitoring and using outcomes, and continuous quality improvement activities.22
2. Patient-to-Provider Ratio	A low participant/provider ratio has been a feature of all tested first episode psychosis services from OPUS which had a caseload per worker of 15 patients.23 The importance of small caseloads is supported in most guidelines.24
3. Services delivered by Team	Multidisciplinary teams have been identified as a core component of successful mental health teams.25 They are a core component of all successful first episode psychosis services, since the time of OPUS up to and including the time of RAISE Navigate.23,26
4. Assigned Case Manager/ Care Coordinator	Assignment of a case manager or care coordinator is an essential component of all successful first episode psychosis services.24 There are many models of case management,27 but the overall effectiveness may depend on the case manager and others delivering specific evidence-based services.28
5. Psychiatrist Caseload	The importance of the frequency of medication visits was demonstrated in the RAISE Navigate study.29 The psychiatrist caseload is, however, a proxy for the detailed best practices of antipsychotic medication prescription.30
6. Psychiatrist Role on Team	Integration of psychiatrists was identified as a core component of successful mental health treatment teams.25 They have been a component of all tested first episode psychosis services.23,26
7. Weekly Multi- disciplinary Team Meetings	Team meetings are essential to the functioning of multi-disciplinary teams providing integrated care. This was first established in assertive community treatment teams.31 They are an essential component of the RAISE Navigate program team fidelity assessment.32
8. Explicit Diagnostic Admission Criteria	The first episode psychosis team treatment model was developed and tested for individuals meeting criteria for a schizophrenia spectrum disorder.23,26 In practice, the services have been made available to both individuals with a first episode of bipolar disorder33 and those at clinical high risk of developing a psychosis.34 The treatment needs and outcomes of those at clinical high risk are significantly different from those with a first episode of a schizophrenia spectrum disorder, and therefore it is necessary to identify the diagnoses of the patients served by the program.
9. Population Served	The annual incidence of schizophrenia is roughly 16:100,000 people. This incidence varies with urbanicity and immigrant status.35 If there is a known, specific annual incidence rate for the population served, that rate should be used rather than a generic incidence rate.36
10. Age Range Served	The age of onset of schizophrenia rises steeply from (pre-)adolescence, to a peak in early adulthood followed by a gradual decline to age 60 years, after which incidence rates level off.37 There is no justification for a cut-off of age 35 years.34,38 Serving the full age range is a National Institute of Health and Care Excellence Quality Standard.17
11. Duration of FEP Program	Randomized controlled studies of first episode psychosis services have focussed on evaluating programs for up to 2 years.2 Three studies comparing longer versus shorter programs and follow-up in usual care found improved outcomes in longer first episode psychosis programs.39–42
12. Targeted Education to Health/Social Service/ Community Groups	Education and outreach have been shown to reduce duration of untreated psychosis.43 A local approach to reducing DUP has also been successful.44 However, a systematic review found too many methodological problems to draw clear conclusions.45
13. Early Intervention	Prior hospitalization is a robust predictor of future hospitalization.46 There is an inconsistent association between duration of untreated psychosis and rehospitalization.47 The RAISE Navigate study, which found an association between DUP and future hospitalization, had a long median DUP of 74 weeks.47 While duration of untreated psychosis is a predictor of long-term outcome,48 it is difficult to measure reliably.49
14. Timely Contact with Referred Individual	Timely access to treatment is important due to high rates of deliberate self-harm50 and violence and aggression before treatment.51 Furthermore, the duration of untreated psychosis is related to outcome.48 In addition, the time to first appointment is related to both attendance and engagement with mental health services.52 Two weeks is the national benchmark in the United Kingdom.17
15. Family Involvement in Assessments	Family involvement in the initial assessment is seen as important for diagnostic assessment53,54 and the engagement of families.55
16. Comprehensive Clinical Assessment	Recommendations on comprehensive clinical assessment can generally be found in clinical practice guidelines.53 Comprehensive clinical assessments include diagnostic and risk assessments.
17. Comprehensive Psychosocial Needs Assessment	The assessment of patient needs, as defined by the patient, can be completed with structured questionnaires56 or by clinician identification of patient goals. Patients have identified working toward their goals as important for maintaining engagement.55
18. Treatment/Care Plan After Initial Assessment	The Joint Commission, which accredits healthcare organizations in the United States, has a standard PC 4.40: “The organization has a plan for care, treatment, or services that reflects the assessed needs, strengths, preferences, and goals of the individual served.” https://www.jointcommission.org › standards. A structured approach to assessing needs and developing collaborative care plans has shown benefit when compared with control condition.57
19. Antipsychotic Medication Prescription	Antipsychotic pharmacotherapy is an essential component of all first episode psychosis treatment services.23,26,58 Antipsychotic pharmacotherapy is effective in reducing symptoms of psychosis and preventing relapse.59–61
20. Antipsychotic Dosing Within Recommendations	Patients with a first episode psychosis respond to lower doses of antipsychotics than those with multi-episode psychosis. In addition to diminishing benefit with higher doses, there are increased side effects.62 A systematic review found no evidence of benefits and increased side effects to antipsychotic doses higher than approved guidelines and to combinations of antipsychotics.63
21. Clozapine for Medication-Resistant Symptoms	In a longitudinal cohort study of 244 first episode psychosis patients, the response rate to a first-line antipsychotic was 75.4% in the first trial, 16.7% in the second trial, but 75% in the third trial with clozapine.64 This suggests that the prevalence of treatment resistance is approximately 25%. In a separate first episode psychosis cohort, median time to clozapine treatment was 42 weeks, which suggests that early psychosis services play a crucial role in the initiation of clozapine.65
22. Patient Psychoeducation	Patient psychoeducation has been described as “systematic, structured, didactic information on the illness and its treatment and includes integrating emotional aspects in order to enable patient or family to cope with the illness.” 66 A systematic review concluded that a course of at least 12 sessions of patient psychoeducation for people with schizophrenia resulted in reduced relapse and hospitalization, greater medication adherence, and self-rated social functioning.67
23. Family Education and Support	Family psychoeducation recognized as an evidence-based practice.68 There is evidence that longer interventions are associated with better outcomes.69 Family psychoeducation combines informational, cognitive, behavioral, problem-solving, emotional, coping, and consultation therapeutic elements.
24. Cognitive Behavior Therapy (CBT)	The effectiveness of cognitive behavioral therapy (CBT) is supported by systematic reviews.70–72 A standardized and widely used definition of a minimal effective dose of psychotherapy is 8 or more sessions.73 A recent health technology assessment found that brief or low intensity CBT at between 6 and 10 sessions was effective, although most studies were in the range of 16 sessions.73,74
25. Supporting Health	Monitoring weight gain and intervening with effective programs can prevent weight gain and achieve weight loss.75 Monitoring of extrapyramidal side effects such as akathisia and tardive dyskinesia can lead to evidence-supported strategies to reduce these side effects.76,77 Detailed assessment of physical health in people with schizophrenia indicates an excess of health problems in this population that could be improved with patient involvement in primary care.78 Glucose and triglyceride abnormalities have been documented in first episode psychosis patients, suggesting that early intervention may be helpful.79 Systematic reviews indicate that pharmacological interventions are effective in smoking cessation.80
26. Annual Formal Comprehensive Assessment	As patients recover from a first episode of psychosis, their levels of symptoms decline, level of functioning improves, and goals change to become more recovery-oriented.81 Clinicians need to adapt to fluctuating clinical presentations. While some funders demand more frequent formal assessments and treatment plans to maintain funding, a documented annual formal review can ensure that progress is monitored.
27. Services for Patients with Substance Use Disorders	A large proportion of patients in first episode psychosis services have substance use disorders and these have a significant negative impact on outcomes. A systematic review of course and treatment of substance use disorders in first episode psychosis found a general positive impact of program participation on reductions in substance use.82 The impact of more specialized services within first episode psychosis services was harder to determine. Results of the RAISE Navigate study indicated low participation in substance use treatment modules and no change in substance use over time.83 The criteria for services in the FEPS-FS are drawn from the Dual Diagnosis Capability in Mental Health Treatment (DDCMHT) scale.84
28. Supported Employment (SE)	A systematic review of employment outcomes in early psychosis programs found an employment rate for supported employment patients of 49%, compared with 29% for patients receiving usual services. The authors concluded that in early intervention programs, supported employment moderately increases employment rates in addition to modest effects early programs alone have on vocational/educational outcomes compared with usual services.85 The rating criteria are drawn from the Individual Placement and Support Fidelity Scale.86
29. Supported Education (SEd)	Supported education adopts principles of supported employment programs and applies them to education.87 Supported education was a component of the Recovery After an Initial Schizophrenia Episode (RAISE) Navigate program.32,88 To date, there have been insufficient well-controlled studies to provide evidence of the effectiveness of supported education in first episode psychosis.87 The RAISE Navigate program combined Supportive Education and Employment (SEE) and reported increased participation in education and work compared with the control condition.89
30. Active Engagement and Retention	Active outreach has been identified as an essential component of first episode psychosis services.9 It has been a component of all the first episode psychosis services tested in randomized controlled trials and fidelity scales based on these trials.90
31. Patient Retention	A systematic review of retention strategies in mental health services has identified several essential components of first episode psychosis effective in increasing retention. These include addressing mental health knowledge, mental health attitudes, and barriers to treatment.91 An epidemiological cohort study demonstrated a 23% dropout rate.92 The FEPS-FS dropout index was developed as a simple ratio that could be calculated by programs using readily accessible data.
32. Crisis Intervention Services	The range of services described varies from the traditional 24-h team care provided by Assertive Community Treatment,31 which is rated as a 5.
33. Communication Between FEP and Inpatient Services	Communication between inpatient and outpatient teams significantly improves the likelihood of outpatient follow-up on discharge.93 Patients who did not have an outpatient appointment at discharge were two times more likely to be hospitalized again in the same year than patients who kept at least one outpatient appointment. The proportion of patients seen within 2 or 4 weeks of hospital discharge is a widely used quality indicator in mental health services and general medical services.94
34. Timely Contact After Discharge from Hospital	The time from discharge to first outpatient appointment was the only health services variable that predicted the likelihood of attending a first follow-up appointment.52 Patients who did not have an outpatient appointment after discharge were two times more likely to be hospitalized in the same year than patients who kept at least one outpatient appointment.95
35. Assuring Fidelity	Successful implementation requires the identification and measurement of the core components of evidence-based practices.96,97 Quality indicators have been developed and applied to compare the quality of first episode psychosis services.98–100 More recently, the Royal College of Psychiatrists of England has applied a set of 8 quality indicators to assess early psychosis intervention programs.101

The use of fidelity scales for self-report has been shown to be potentially reliable,²⁰ and FEPS-FS has been used for the assessment of broad trends in practice.^5,21 The FEPS-FS 1.0 includes a guide, based on the structured interview guide to help teams conduct a self-assessment. The reliability of this approach has not been tested.

Discussion

The FEPS-FS 1.0 is a significant improvement on the original FEPS-FS and can now be used reliably for remote assessments. The reliability of onsite assessments has not been tested for FEPS-FS 1.0, but it was high in the original version.⁸ The modifications to improve reliability of the self-report version have not been tested. The same scale, interview guide, and data are used for onsite, remote, and self-report fidelity assessments. The scale differentiates among programs with high, acceptable, and poor fidelity and can assess and compare programs which use different models for service delivery training and support.¹⁰

The modifications to the scale represent pragmatic changes to the existing framework rather than fundamental changes in response to a changing evidence base. For example, monitoring health indicators was moved from a generic annual review component to a separate supporting health component that included engagement in primary care. This change fitted the pragmatics of who was accountable for components of care and where the data could be found in the healthcare record.

The scale uses the sum of the unweighted items to compute a total score. Both individual item scores and total scores can be used to compare programs and to measure success in implementation studies or quality improvement initiatives. Total scores and component scores can be summed to assess the quality of care across networks of programs. For example, a low score across a component such as supported employment may be related to large-scale funding for supported employment services or it may reflect problems in staff training or other aspects of implementation.

The review method used in this study has limitations. The studies that used the scale were not designed to assess the fidelity scale, and the evaluations of the scale were secondary outcomes of the studies. We did not conduct new systematic reviews of each of the components; rather, we identified up-to-date reviews and guidelines such as the National Institute for Health and Care Excellence to support specific components.¹⁷ The review method is appropriate for the purposes presenting an update, but it does not represent a fundamental review of the foundations of the scale.

Future studies should be designed to assess the predictive validity of the scale. This will require an adequate number of programs with a range of fidelity scores and high rates of completion of reliable outcome data. This should be feasible within the EPINET programs.¹⁰² New systematic reviews of each of the components could be used to modify existing components or identify new components that have both strong evidence of effectiveness and efficacy. Finally, the application of the scale needs to be broadened to include services for patients who are at clinical high risk of psychosis and those with bipolar disorder.