Family-focused therapy for individuals at clinical high risk for psychosis: treatment fidelity within a multisite randomized trial

Abstract

Aim

Family psychoeducation is an effective adjunct to pharmacotherapy in delaying relapse among patients with schizophrenia and bipolar disorder. This study tested the treatment adherence and competence of newly trained clinicians to an adaptation of family-focused therapy for individuals at clinical high risk for psychosis (FFT-CHR).

Methods

The sample included 103 youth or young adults (ages 12–30 years) who had attenuated positive symptoms of psychosis. Families participated in a randomized trial comparing two psychosocial interventions: FFT-CHR (18 sessions over 6 months) and enhanced care (EC; 3 sessions over 1 month). Following a 1.5-day training seminar, 24 clinicians from eight study sites received teleconference supervision in both treatment protocols for the 2-year study period. Treatment fidelity was rated with the 13-item Therapy Competence and Adherence Scales, Revised.

Results

Supervisors classified 90% of treatment sessions as above acceptable fidelity thresholds (ratings of 5 or better on a 1–7 scale of overall fidelity). As expected, fidelity ratings indicated that FFT-CHR included a greater emphasis on communication and problem-solving skills training than EC, but ratings of non-specific clinician skills, such as maintaining rapport and appropriately pacing sessions, did not differ between conditions. Treatment fidelity was not related to the severity of symptoms or family conflict at study entry.

Conclusions

FFT-CHR can be administered with high levels of fidelity by clinicians who receive training and supervision. Future studies should examine whether there are more cost-effective methods for training, supervising and monitoring the fidelity of FFT-CHR.

INTRODUCTION

Individuals who are at ‘clinical high risk’ (CHR) for psychosis experience attenuated positive symptoms such as abnormal perceptual experiences and problems related to social and role functioning.¹ Longitudinal research indicates that approximately 35% of CHR individuals convert to psychosis within 2.5 years.² Successful ascertainment has spurred the development of early interventions to delay or prevent psychosis and to enhance functioning.

Family-focused therapy (FFT) is a manualized psychoeducational treatment that is an effective adjunct to pharmacotherapy in stabilizing the course of bipolar disorder in adults and adolescents.^3–6 In FFT, clinicians provide psychoeducation and teach communication and problem-solving skills. FFT aims to decrease family conflict and criticism and enhance positive family involvement. Positive family involvement predicts decreased symptoms and enhanced social functioning in CHR youth and in youth with mood disorders.⁷

Recently, we modified the FFT manual to include a range of psychoeducational materials more relevant to the symptomatic characteristics of CHR individuals. Specifically, we emphasized behavioral activation techniques to address negative symptoms and the use of communication clarity exercises to reduce communication deviance.^8,9 The current study evaluates therapist fidelity to treatment in a randomized controlled trial (RCT) of FFT-CHR. Treatment fidelity refers to the extent to which the essential features of an intervention are delivered as prescribed by a treatment protocol^10–12 and requires careful attention to intervention design, training of clinicians and ongoing monitoring of intervention delivery.¹⁰ Typically, intervention delivery is evaluated by expert raters using an observational fidelity measure.^10–12 Examination of treatment integrity is particularly important when the treatment and/or population is novel and there is no precedent for how much symptom change should be expected from a well-implemented intervention, as is the case with emerging research on family treatment with CHR youth.

A review by Perepletchikova et al.¹¹ found that systematic evaluation of treatment integrity occurs in only 3.5% of published psychotherapy trials. When fidelity monitoring reveals that a treatment did not deliver essential content as specified in the manual, results may not be interpretable and the trial may not represent a fair test of the treatment. For example, in a trial of individual cognitive behavioral therapy (CBT) with CHR individuals, Addington and colleagues¹³ found similar improvement in positive symptoms, depression and anxiety for CBT and supportive therapy treatment groups. Because the authors included careful manipulation checks on adherence to treatment delivery, including therapists’ post-session reports of per cent time engaged in core CBT techniques, they were able to explore explanations for the lack of differential efficacy, including the possibility that CBT therapists did not provide an adequate dose of CBT. Among the 53 studies of family intervention for schizophrenia included in Pharoah et al.’s¹⁴ comprehensive review, only one, the NIMH Treatment Strategies in Schizophrenia collaborative study, provided information regarding systematic assessment of fidelity during a large-scale treatment trial.¹⁵

The current study utilized an observational fidelity measure, the Therapy Competence and Adherence Scales, Revised (TCAS-R),^6,16 to examine clinician fidelity to FFT-CHR and enhanced care (EC). Ratings of clinician behaviors and techniques on the TCAS-R were used to evaluate whether the two treatment conditions were implemented according to manualized protocols. First, we expected clinicians in both conditions to achieve high levels of adherence and competence in the majority of sessions because of the regular (i.e. biweekly) supervision and the well-operationalized FFT-CHR and EC protocols. We hypothesized that clinicians administering FFT-CHR and EC would demonstrate equivalent levels of fidelity with regard to: (i) general therapeutic factors such as building rapport with all participants; and (ii) administration of the specific psychoeducational aspects of the therapy that are included in both protocols. However, only the FFT-CHR manual included communication and problem-solving training, so we expected there would be significant differences between FFT and EC in the frequency with which these skills were addressed during treatment.

An exploratory aim was to investigate whether the pretreatment severity of CHR patients’ symptoms or family conflict offered unique challenges to successful implementation of FFT-CHR or EC. We hypothesized that both treatments were well targeted to the typical presenting symptoms of individuals in this population and flexible enough to be utilized competently with families of varying conflict levels. Thus, we did not expect there would be strong associations between ratings of clinician fidelity and CHR individuals’ symptoms or family conflict.

METHOD

Participants

Study participants were a subset of individuals from the North American Prodrome Longitudinal Study (NAPLS). Participants were between the ages of 12 and 35. They met inclusion criteria for the NAPLS study if they had one of three prodromal risk syndromes based on the Structured Interview for Prodromal Syndromes: (i) attenuated positive symptoms: patients are experiencing positive symptoms (unusual thoughts, suspiciousness, grandiosity, perceptual disturbances, disorganized communication) that are sub-psychotic in duration and intensity that have begun or worsened in the past year; (ii) brief intermittent psychosis: patients are experiencing fully psychotic symptoms that are present only intermittently with onset in the past 3 months; (iii) genetic risk and deterioration: patients either have schizotypal personality disorder or have a first degree relative with a psychotic disorder and have experienced a significant decline in functioning in the last year.^17–19 Participants were excluded from the study if they met criteria for an Axis I schizophrenia-spectrum diagnosis, a neurological disorder, current drug or alcohol dependence, or IQ below 70.

Between January 2009 and February 2012, a total of 129 NAPLS subjects expressed a willingness to participate in a randomized clinical trial of family therapy. After informed consents and/or assents were obtained from all participants, families were randomized to FFT or EC treatment.

Treatment conditions

There were three modules included in the FFT treatment, each of which involved approximately six sessions, for a total of 18 sessions over 6 months. Sessions were conducted weekly during the first 3 months of treatment and then every 2 weeks during the last 3 months. During the first module, psychoeducation, the clinician facilitated a family discussion of symptoms, explained the vulnerability stress model, and encouraged family members to identify triggers for symptoms and effective coping strategies. During the second module, the clinician taught the family-adaptive communication skills such as expressing positive feelings, active listening and making a positive request for change, and during the last module families were taught structured problem-solving techniques. Family members actively practiced communication and problem-solving skills with one another during sessions and were asked to practice between sessions.

The comparison treatment, EC, consisted of three family psychoeducation sessions conducted once per week. The EC treatment was a streamlined version of the FFT psychoeducation module. Clinicians were instructed not to teach communication and problem-solving skills during EC treatment. The same therapists provided FFT and EC.

Clinician training and supervision

A 1.5-day training workshop was conducted by Drs Miklowitz, O’Brien, Cannon, Schlosser, De Silva and Jamie Zinberg, MA, at UCLA prior to initiation of the study, and included clinicians from each of the eight participating sites (Emory University, Harvard University, University of Calgary, University of California Los Angeles, University of California San Diego, University of South Carolina, Yale University and Zucker Hillside Hospital). Twenty-four masters- and doctoral-level psychologists were trained to provide both the FFT-CHR and EC treatments and were supervised by Drs Miklowitz, O’Brien, De Silva or Jamie Zinberg, MA. All treatment sessions were videotaped or audiotaped. For the first FFT-CHR and EC case that a clinician treated, the supervisor viewed at least every other session and provided prompt feedback. Clinicians received at least an hour of individual supervision every 1–2 weeks.

Measures

Fidelity ratings

One clinical supervisor (DJM) trained all of the clinical supervisors to conduct TCAS-R ratings. Another clinical supervisor (MO) was considered the master rater for the study and contributed approximately 70% of all rated sessions. The three secondary raters contributed the remaining 30%. Clinicians did not know which therapy session would be randomly selected for formal ratings by supervisors. For each family participating in FFT, three randomly selected sessions were rated. One session was selected from each of the three treatment modules. For each family participating in EC, one session was randomly selected and rated.

The TCAS-R consists of 13 items rated on 7-point scales (see Table 1). The first item, psychoeducation, focuses on the psychoeducational treatment elements that are prescribed in both the FFT and EC manuals. Six scales address prescribed interventions that are unique to FFT (communication training, role playing, giving and soliciting feedback, problem solving, problem specification and homework). Three scales focus on general clinical skills, which are relevant to both conditions (rapport, pacing, session command). One scale, overall rating, provides an overall summary rating of fidelity. Two scales in the original TCAS were not included in the present study.

TABLE 1.

TCAS item description and interrater reliabilities

TCAS primary codes		ICC
Prescribed techniques in FFT and EC
Psychoeducation	Provides accurate information in an interactive style that is personalized to the family/patient	0.73
Prescribed in FFT
Communication	Provides specific instruction on communication skills	0.86
Role playing	Coaches the family in role-playing communication skills	0.78
Feedback	Provides feedback that is constructive and predominantly positive	0.84
Problem specification and problem solving	Facilitates behavioural problem-solving strategies for identifying problems, generating solutions and planning for follow-through	0.78†
Homework	Homework is assigned with clear rationale and directions	0.61
Clinician skills
Rapport	Conveys empathy, warmth and genuineness, and connects meaningfully with each participant	0.49
Pacing	Structures session, controls flow of discussion, limits unproductive discussion	0.65
Session command	Is directive and focused on planned agenda determined by collaborative goal setting	0.70
Clinician overall	Overall delivery of prescribed techniques and general skills	0.67

^†Reliability for these items was calculated using agreement on account of low base rate and a restricted range.

EC, enhanced care; FFT, family-focused therapy; ICC, intraclass correlation; TCAS, Therapy Competence and Adherence Scales.

The 7-point TCAS-R scales range from 1 (very poor) to 7 (excellent); items can also be rated as 0 (not applicable). Scales capture both whether a skill was delivered (adherence) and how skillfully this was done (competence). Ratings from 1 to 3 indicate a lack of adherence to the manual. Ratings from 4 to 7 indicate that the therapist adhered to the manual, with higher scores reflecting higher competence and skill in tailoring the material to the needs of patients.

The TCAS has demonstrated interrater reliability in an efficacy study of FFT for bipolar disorder.⁶ For the current study, a total of 32 sessions from both treatment conditions (25 from all phases of FFT-CHR; 7 from EC) were rated independently by trained raters. Intraclass correlation coefficients for the 11 TCAS scales were calculated by comparing the master rater (MO) with all secondary raters.

Family conflict

Family conflict was measured with the Conflict Behavior Questionnaire (CBQ) short form, mother report. The CBQ consists of 20 true/false items that capture caregivers’ perceptions of conflict and has demonstrated criterion-related validity, internal consistency and test–retest reliability.²⁰

Attenuated symptoms

Positive and negative symptoms were rated on the Scale of Prodromal Syndromes (SOPS)¹⁹ from 0 (absent) to 6 (psychotic) and were summed to create a total symptom score.

Data analysis

The primary study questions were examined by defining three independent variables: treatment condition (FFT vs. EC), family conflict (mothers’ CBQ ratings) and symptom severity (summed positive and negative symptoms on SOPS). The dependent variables were the 11 TCAS items. We used one-way analyses of variance (ANOVAs) to test the hypothesis that: (i) the quality of the psychoeducation provided by clinicians was not significantly different between treatment conditions; and (ii) therapists’ general skills (i.e. rapport, pacing and session command) were not significantly different across conditions. A separate one-way ANOVA examined whether the overall rating of treatment fidelity was significantly different across treatment conditions.

Using χ² statistics, we compared the occurrence of communication skills training and the occurrence of problem-solving training in FFT-CHR versus EC sessions. To test our prediction that neither the severity of patient symptoms (total SOPS score) nor the level of family conflict would be significantly associated with ratings of fidelity across the two treatment conditions, we conducted analyses of covariance (ANCOVAs) assessing the comparability of the overall fidelity rating scale across conditions while covarying total symptoms and family conflict.

RESULTS

A total of 129 participants and families were randomized, 66 to the FFT-CHR condition and 63 to the EC condition. Eleven families dropped out of FFT-CHR and nine dropped out of EC before initiating treatment. Thus, a total of 109 families initiated treatment. There were no recorded sessions available for six participating families due to technical difficulties with video- or audiotaping. Thus, TCAS ratings were available for 103 families (EC = 50, FFT = 53), on a total of 172 session recordings.

We aimed to document three randomly selected ratings (one per module) for each course of FFT (N = 53). However, every family did not participate in all 18 FFT-CHR sessions and therefore analyses were based on 122 FFT sessions. In the EC condition, one rating was available for each of the 50 families.

The mean age of youth participants was 17 years; 39% were female; 60% (n = 62) self-identified as white/Caucasian, 12% (n = 12) as African American, 5% (n = 5) as Central/South American, 4% (n = 4) as Native American, 3% (n = 3) as South Asian, 1% (n = 1) as West/Central and Middle Eastern, 1% (n = 1) as Pacific Islander, 13% (n = 13) as multi-racial, and missing for 1%. Treatment included a CHR individual and his/her parent(s) and, where possible, sibling(s). In three cases, treatment included a CHR participant and a partner/spouse.

The mean reliability (intraclass correlation) coefficient for the 11 TCAS item ratings was 0.71 (32 cases). The majority of items demonstrated good reliability (Table 1), with coefficients that were higher on average for prescribed techniques (psychoeducation, communication strategies, problem-solving strategies, homework) (mean ICC = 0.77) than for non-specific factors (rapport, pacing, session command) (mean ICC = 0.61). The overall rating, a summary evaluation of how well the clinician administered the treatment during the session, had acceptable reliability (0.67, P = 0.001). For problem specification and problem solving, which had a restricted range of scores, reliability was estimated based on per cent agreement on ratings of greater than or equal to 5 versus those less than or equal to 4. Reliability for these two items was 0.78.

Mean ratings of fidelity were well within the competent range for the 11 TCAS scales, with a range from 5.56 for assigning homework to 6.04 for therapist command of sessions. The overall rating of TCAS fidelity for clinicians in 46 (92%) of the 50 EC sessions and 109 (89%) of the 122 rated FFT sessions was 5 or higher, which is equivalent to ‘good’ or better. The mean overall rating of fidelity did not differ between FFT (M = 5.85) and EC conditions (M = 5.78); F(1, 170) = 0.21, P = 0.65.

A one-way ANOVA showed that clinicians were equally skillful in providing psychoeducation in the FFT-CHR and EC conditions, F(1, 136) = 0.03, P = 0.87. There were also no significant differences in ratings of therapist general skills, including the quality of rapport established with families, F(1, 170) = 0.28, P = 0.60, pacing of the sessions, F(1, 170) = 0.36, P = 0.55, and command of the sessions, F(1, 170) = 0.16, P = 0.69.

Clinicians provided communication enhancement training significantly more often in FFT than in EC sessions, χ²(3, N = 172) = 17.36, P = 0.001. Problem-solving strategies were also delivered more frequently during FFT compared with EC treatment, χ²(3, N = 172) = 7.27, P = 0.03.

Data on family conflict as reported by patients’ mothers were available for 68 of the 103 families (66%). A one-way ANCOVA indicated that overall fidelity (based on the item overall rating on the TCAS-R) was equivalent across treatment conditions, F(1, 65) = 0.26, P = 0.62, when the level of conflict in the family was covaried, F(1, 65) = 0.43, P = 0.51. There was no main effect for treatment condition, F(1, 99) = 0.27, P = 0.61, on overall ratings on the TCAS-R when total symptoms at the commencement of treatment were covaried, F(1, 99) = 0.62, P = 0.43. The zero-order correlations between SOPS total symptoms score and fidelity ratings for the psychoeducation (r = 0.09, P = 0.3), communication (r = 0.05, P = 0.80) and problem-solving skills training (r = 0.04, P = 0.78) modules were all non-significant.

DISCUSSION

We evaluated fidelity in an eight-site randomized trial of FFT-CHR that included: the use of two distinct treatments that are well specified in treatment manuals, a 1.5-day initial workshop, ongoing supervision of clinicians, and careful monitoring of treatment adherence and competence. FFT and EC conditions did not differ significantly with regard to general therapeutic factors such as building rapport during sessions, and therapists’ provision of psychoeducation, but were significantly different in the use of communication and problem-solving training. These findings indicate that the treatments were delivered as planned. Most of the sessions in both conditions (89–92%) were rated as ‘good’ or better in terms of therapists’ provision of adherent and competent treatment. These results are similar to those of family intervention trials that have used earlier versions of the TCAS,^4,16 and suggest that with training and ongoing supervision, clinicians with varying backgrounds and levels of previous training in family therapy can implement two distinct psychoeducational treatments, build rapport with families, skillfully direct sessions and pace them appropriately. Whether high fidelity can be obtained with less intensive training and supervision remains to be evaluated.

Ratings of clinician fidelity were not strongly influenced by family conflict or symptom severity. A conflictual interaction style among family members may pose challenges to the therapist, but the structure of FFT-CHR and EC may help contain a family’s tendency towards negative content and processes. Specifically, in the FFT-CHR condition, skills are taught in a stepwise fashion, which allows the family to practise new strategies with relatively non-conflictual topics before engaging with highly charged problems. Psychoeducation, communication and problem-solving techniques are tailored to the unique needs of each family and this flexibility within the FFT model allows for a broad range of symptoms to be addressed.

A limitation of this study includes the low interrater reliabilities of some TCAS-R items (e.g. homework, rapport). Restriction of range on these TCAS-R items may have deflated reliability scores. Although this was an RCT with ‘blind’ evaluations of clinical outcome, the clinical supervisors knew whether they were rating FFT-CHR or EC sessions. We did not obtain independent ratings by an ‘uninterested’ third party.

In implementation studies examining the effectiveness of evidence-based psychosocial interventions in community care settings, alternative methods for assessing fidelity may be needed to streamline monitoring in diverse settings. Promising alternatives may include obtaining measures of fidelity from clinicians themselves^21,22 or establishing benchmarks for quality of therapists’ delivery of a specific treatment before being able to practise without supervision. The successful community implementation of evidence-based treatments will rest in part on whether cost-effective protocols for training, supervising and monitoring the fidelity of clinicians can be developed.