Family-based treatment for children and adolescents with eating disorders: a mixed-methods evaluation of a blended evidence-based implementation approach

Abstract

In this study, we evaluated a blended implementation approach with teams learning to provide family-based treatment (FBT) to adolescents with eating disorders.

Four sites participated in a sequential mixed method pre–post study to evaluate the implementation of FBT in their clinical settings. The implementation approach included: (a) preparatory site visits; (b) the establishment of implementation teams; (c) a training workshop; (d) monthly clinical consultation; (e) monthly implementation consultation; and (f) fidelity assessment. Quantitative measures examining attitudes toward evidence-based practice, organizational learning environment and organizational readiness for change, as well as, individual readiness for change were delivered pre- and postimplementation. Correlational analyses were used to examine associations between baseline variables and therapist fidelity to FBT. Fundamental qualitative description guided the sampling and data collection for the qualitative interviews performed at the conclusion of the study. Seventeen individuals participated in this study (nine therapists, four medical practitioners, and four administrators). The predetermined threshold of implementation success of 80% fidelity in every FBT session was achieved by only one therapist. However, mean fidelity scores were similar to those reported in other studies. Participant attitudes, readiness, and self-efficacy were not associated with fidelity and did not change significantly from pre- to postimplementation. In qualitative interviews, all participants reported that the implementation intervention was helpful in adopting FBT. Our blended implementation approach was well received by participants. A larger trial is needed to determine which implementation factors predict FBT fidelity and impact patient outcomes.

Implications.

Practice: A larger trial is needed in order to continue to study our implementation model to further enhance adoption of family-based treatment (FBT) in clinical practice for children and adolescents with eating disorders, along with an evaluation of clinical outcomes.

Policy: Policymakers should provide time and funding for ongoing clinical consultation for seasoned clinicians, as well as training for new staff in family-based treatment.

Research: Practitioners desire ongoing support and collaboration in sustaining fidelity to family-based treatment.

BACKGROUND

Implementation science is the process of systematically investigating methods that improve the uptake and use of evidence-based treatments (EBTs) into routine clinical practice [1–3]. Implementation research improves upon the typical “train and hope” approach to implementing EBTs that relies solely on support from didactic workshops and educational materials. This past approach may result in knowledge improvement, but little to no change in practice [4–6]. Implementation efforts within the field of behavioral health care may lag behind more traditional areas of medicine and face unique challenges in the field of complex children’s mental health [7]. Considering that eating disorders are one of the most complex disorders that present in childhood and have the highest mortality rate of any psychiatric illness, implementation efforts in this area are sorely needed.

The present study combined elements of existing implementation frameworks to train four Ontario behavioral health provider organizations in an EBT called family-based treatment (FBT) for children and adolescents with eating disorders. Our blended implementation approach was evaluated using qualitative and quantitative methodology. The frameworks used included a process framework, the active implementation frameworks (AIF) [8], a determinants framework, consolidated framework for implementation research (CIFR) [9] and an evaluation framework, implementation outcomes taxonomy (IO) [10].

Implementation challenges

There is little published implementation research in the field of eating disorders generally, and even less for children and adolescents more specifically. Some work has been done by Waller [11] on therapist drift in cognitive behavioral therapy (CBT) for adults with eating disorders. In the pediatric population, implementation research has focused on identifying barriers and facilitators of FBT implementation [12] and on developing a knowledge transfer framework for FBT implementation [13]. We are not aware of any study that has explicitly adopted and evaluated a model of EBT implementation within the eating disorders field.

Family-based treatment

The treatment of children and adolescents with eating disorders is complex, and a body of research has developed on evaluated psychotherapeutic approaches [14–17]. Among those treatments shown to be effective is FBT, an outpatient, intensive treatment in which the family is the primary resource to re-nourish the affected child [18]. Family-Based Treatment is a manual-guided intervention that involves three phases of treatment delivered over 9–12 months. The first phase focuses on helping the family to restore the child’s weight and interrupt disordered eating behavior. The second phase involves the transition of control over eating behavior back to the adolescent. The third and final phase addresses developmental issues such as physical development, peers and dating, and separation and individuation. Family-Based Treatment requires a therapist to deliver the psychotherapy and a medical practitioner to evaluate the physical health of the young person at the outset and to monitor for medical complications throughout the treatment.

Family-Based Treatment is empirically supported by one meta-analysis demonstrating superiority over individual treatment in terms of long-term outcomes [19], and two high-quality randomized controlled trials demonstrating superiority over individual [20], and other family psychotherapies [21]. FBT has the potential to reduce treatment costs by up to 70% due to reductions in hospitalization [22]. Despite this, few therapists deliver FBT with fidelity in their clinical settings [12]. Treatment fidelity in clinical delivery is important for replicating outcomes, particularly those published in research trials [23]. Thus, improving FBT implementation, delivery, and fidelity in clinical settings is necessary in order to optimize outcomes for this patient population.

Frameworks informing our implementation approach

The implementation approach for this study was informed by three frameworks. Firstly, the AIF [24], a process framework that characterizes the overarching phases of implementation. Secondly, the CFIR, a determinant framework [25] detailing factors associated with successful implementation. And finally, the IO, an evaluation framework that identifies several implementation outcomes as distinct from clinical or system outcomes [10]. These frameworks have been used to inform the successful adoption and implementation of EBTs within the mental health sector [26–31].

The AIF describes six stages of implementation, including (a) exploration and adoption, (b) program installation, (c) initial implementation, and (d) full implementation. Briefly, the Exploration and Adoption stage is the point at which the implementing organization considers the “fit” between the proposed EBP and the practice environment and readiness to implement the EBP model into routine clinical practice. During the second and third stages of implementation—Program Installation and Initial Implementation—the organization conducts training and makes structural changes to accommodate the new intervention. The fourth stage of implementation refers to Full Implementation, where the EBT becomes a designated model of delivery and is used in everyday practice. A key innovation of the AIF is the use of implementation teams who are situated within the implementing organization to oversee and monitor the implementation process and devise procedures and protocols to support EBT implementation in everyday practice.

The CFIR identifies 39 evidence-based constructs that are associated with successful EBP implementation, organized within 5 overarching domains: (a) intervention (EBP) characteristics, (b) the outer setting, (c) the inner setting, (d) clinician characteristics, and (e) the implementation process. Evidence for the applicability of the CFIR framework for pediatric eating disorders is supported by evidence from the general mental health and addictions fields [30–34]. These settings mirror those of pediatric eating disorder treatment programs in their complex organizational structures and the type of clinician education and skill set [35]. Similarly, organizations that have used the CFIR to measure and facilitate implementation endeavors are similar to pediatric eating disorder programs in their multilevel leadership and decision-making structures [34], the need for champions of change [33], and consistent supervision and coaching [32].

The IO framework describes eight outcomes that are specific to implementation research including acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability [10]. We examined acceptability and fidelity within our study. Based on the available evidence for the AIF, CIFR, and IO, our implementation model included: (a) preparatory site visits; (b) the establishment of implementation teams; (c) a training workshop; (d) monthly clinical consultation; (e) monthly implementation consultation; and (f) fidelity assessment. In the present study, we evaluated the effectiveness of our blended implementation approach for FBT in four pediatric eating disorder treatment programs that are part of a provincial network of government-funded specialized eating disorder services in Ontario, Canada [36].

METHODS

The central research question was addressed using a sequential explanatory mixed methods design. Quantitative data were dominant, informing qualitative data collection. Integration of the quantitative and qualitative strands occurred during the interpretation phase of the study. Reporting was informed by the Standards for Implementation Research (StaRI) standards (Appendix 1) [37–39]. Ethical approval was received from the Hamilton Health Sciences/McMaster Faculty of Health Sciences Research Ethics Board, as well as the ethics boards at each of the participating sites.

Setting

Clinicians from four sites were recruited after expressing interest in further training in FBT during our previous qualitative work [12]. One site is an academic health science center located in a large urban area. This program has inpatient, day hospital, and outpatient components. The other three sites are community-based behavioral health provider organizations that provide only outpatient services.

Participants

A purposeful sampling approach was used to identify the organizations participating in this study. We recruited agencies that: (a) belonged to the Ontario Community Outreach Program for Eating Disorders, (b) provided psychotherapeutic interventions to children and adolescents (<18 years old) diagnosed with eating disorders, (c) were interested and willing to have staff trained in, deliver, and be monitored in their delivery of FBT.

A total of 17 individuals at the 4 organizations participated in this implementation study (6 individuals at site 1, 3 individuals at Site 2, 4 individuals at Site 3, and 4 individuals at Site 4). Three participants self-identified as male, and 14 as female. Participants had an average age of 46.7 ± 10.5 years (range 28–60 years) and had been in their current role for an average of 7.9 ± 7.0 years (range 1 month to 25 years). Nine participants were therapists, four were medical practitioners (Medical Doctors [MDs] or Nurse Practitioners [NPs]), and four were administrators. Sixteen participants attended the training workshop, and all participated throughout the study. The participant who could not attend the workshop was an administrator. There was no loss to follow-up in terms of collecting measures and final interviews. One therapist retired during the study, but he completed the therapy with his patients and completed all fidelity assessments and study measures. All therapists submitted fidelity assessments and completed their final study measures. Eligible children, adolescents, and their families were those where the child had been newly diagnosed with an eating disorder (first episode) and had not yet received FBT.

It is important to note that there are no definitive sample size requirements for qualitative studies employing fundamental description. Rather, recent methodological guidelines suggest that the sampling approach and size should be directed by: (a) the extent of data collected from participants, (b) whether or not the collected data reliably addresses the study’s research questions, (c) the quality of the data collected, and (d) the analytical strategy to be employed. Given that our explicit emphasis was to describe the perceptions and experiences of a blended implementation model to support FBT use within pediatric eating disorder treatment settings in Ontario, and that we collected multiple types of data from these purposefully selected organizations and their representative staff, the homogeneity of our sampling approach and extensiveness of our data collection processes support the size of the sample selected [40, 41].

Implementation strategy and intervention

Preparation

Therapists, physicians, and administrators from four Ontario-based pediatric eating disorder programs were purposefully recruited to participate in training, clinical and implementation consultation, and evaluation using qualitative interviews and quantitative measures. Each participating organization was invited to form an implementation team [8] consisting of an administrator/manager, a lead therapist, and a medical practitioner, who together, would support FBT training, supervision, implementation, and research processes. Each implementation team was asked to identify therapists in their program who were most appropriate and willing to participate in the study.

In keeping with best practice suggested by the AIF [8], Phase 1 of implementation involved site visits by two authors (J.C. and M.K.) for the purposes of engaging participation and assessing fit for FBT implementation. These coauthors ensured readiness and commitment to the study by discussing all elements of the study with the implementation team, asking each individual if they were willing to participate, and by obtaining a signed consent for the study at the time of the site visit.

Training

Administrators, therapists, and medical practitioners from the four sites attended an in-person two-day training workshop in a central location. Information on implementation models and success was presented in the first hour in a didactic fashion by two of the coauthors (M.K. and J.C.). This was followed by another hour of didactic instruction on the role of the medical practitioner, provided by a coauthor (S.F.). Standard training in the FBT model was then provided by another coauthor (J.L.), lasting one and a half days. Most of this training was didactic but did include some role-play. Evaluation of the training occurred through individual qualitative interviews at the conclusion of the study.

Clinical consultation

Therapists were invited to enroll one or two families in FBT and audio-record each session. Monthly 1-h group clinical consultation calls were provided by telephone by a coauthor (J.C.) for each site. Feedback was provided to participants based on fidelity assessments of submitted audio recordings as well as a discussion of the details of the case with the therapists. Thirty-five clinical consultation sessions were conducted in total, with an average of 9 sessions per site (range 7–12). Consultation calls ranged in length from 4 min to 58 min, with an average of 39 min. Participants were supported by consultation calls for 11 months (range 7–15 months). Each therapist received an average of 7.6 clinical consultation calls (range 5–11).

Implementation consultation

Monthly implementation consultation calls were provided concurrently to the clinical consultation calls by two coauthors (J.C. and M.K.). These calls were allotted 1 h of time and involved the implementation team (administrator, medical practitioner, and lead therapist). Twenty-five implementation consultation calls were conducted with an average of 6 sessions per site (range 4–9). The calls ranged in length of time from 15 min to 52 min, with an average of 27 min. Implementation consultation was provided over a period of 9.5 months (range 7–12 months). Attendance on the calls was as follows: lead therapist 96% (24/25), medical practitioner 48% (12/25), and administrator 64% (16/25).

Fidelity assessment

The primary implementation outcome was therapist fidelity to FBT [10]. FBT implementation success was operationally defined a priori as 80% of therapists at each site demonstrating 80% (average 5.6/7 or greater average score) fidelity to the model at each session. Fidelity of each audio-recorded therapy session was rated by an FBT expert (J.C.) using a validated FBT fidelity measure described below (although at the time of the conception our study, this scale was in development) [42]. Fidelity was also self-rated by the therapists, and parents also rated fidelity immediately after each session. Each therapist recorded one case at minimum and submitted Sessions 1–3, from Phase 1, with one additional session from Phase 1, along with two sessions from Phase 2, and one session from Phase 3 for expert rating.

Intervention

Family-based treatment (FBT) is a manualized, outpatient, intensive treatment with three phases [18]. The first phase focuses on empowering parents to take charge of eating disorder symptoms in order to restore the child’s weight and interrupt disordered eating behaviors. The second phase involves the transition of control over eating and exercise, back to the adolescent. The third and final phase addresses developmental issues such as physical development, peers and dating, and separation and individuation. There are markers for moving from Phase 1 through to Phase 3 that should be attained and there is flexibility on the part of the therapist to make judgments on whether the family is ready to move through these phases. The medical practitioner sees the patient regularly throughout FBT with a frequency that he/she deems medically necessary.

Quantitative measures

FBT fidelity.

Fidelity to FBT was the primary implementation outcome assessed in the present study based on an FBT fidelity instrument developed by Forsberg et al. [42]. Item responses are on a 7-point Likert scale from “1” (“not at all”) to “7” (“very much”), with higher scores indicative of greater use of the FBT components in a given session. For the purposes of the present study, we report on expert ratings of FBT fidelity. A companion paper will report on the concordance of FBT fidelity ratings between expert, therapist self-rating, and parental ratings.

Readiness, attitudes, and self-efficacy.

We collected quantitative measures at baseline and at the study conclusion. Measures included a modified version of the short form of the Organizational Readiness for Change Scale (ORC) [43], the most widely used measure of organizational readiness [44]. The ORC includes 115 items rated on a 5-point scale. Organizational learning climate was assessed using the organizational learning survey (OLS) which includes 18 items rated on a 7-point scale [45]. Clinician readiness for change was assessed using the brief individual readiness for change scale (BIRCS) which includes five items rated on a 5-point scale [46]. Participant attitudes toward EBP were assessed using the evidence-based practice attitudes scale (EBPAS) which involves 15 items on a 5-point scale [47, 48]. Finally, an adapted version of the perceived attributes of the principles of effectiveness (MPAQ) [49] measure was used to elicit therapists' perceptions of the FBT model fit with their program. This measure has 17 items, rated on a 5-point scale. These measures captured elements of the CIFR framework including: organizational readiness (ORC), tension for change (ORC), networks and communication (ORC), relative priority (ORC) and organizational culture (ORC), clinician readiness for change (BIRCS), clinician and administrator attitudes about EBTs (EBPAS), learning climate (OLS), relative advantage (MPAQ), trialability (MPAQ), and complexity (MPAQ).

Quantitative data analysis

Given the exploratory nature of this work, our primary outcome (implementation success) was assessed using descriptive statistics. The percentage of therapists demonstrating FBT implementation success was determined by using the FBT fidelity rating scale, achieving an average score of 80% (5.6/7 or greater) on each treatment session. To explore the extent to which fidelity was associated with implementation factors previously investigated in the literature, we also conducted correlational analyses. We correlated the total fidelity raw score from Sessions 1–3 with baseline demographic characteristics and measures of organizational readiness (ORC), OLS, BIRCS, attitudes toward evidence-based practice (EBPAS), and perceptions of FBT fit (MPAQ). We also examined pre–post scores on these measures and compared them using paired t-tests.

Qualitative measures

Qualitative interviews were used to explore therapist, medical practitioner, and administrator experiences of the blended implementation model. Interviews focused on their perceptions of (a) the overall execution of the implementation model, (b) the overall success of the implementation, and (c) the effectiveness of each component of the implementation model. Interviews were conducted postimplementation, and were audio-recorded and transcribed verbatim. Audio-recording and transcription were guided by the principles of fundamental qualitative description [50].

Qualitative data analysis

In order to generate a description of the participants’ perceptions and experiences of implementation, we used conventional content analysis [51]. Key concepts were identified through line-by-line reading and categorizing of text within the transcripts. A codebook was generated and refined through multiple readings of the transcripts, in consultation with the research team, as well as through the process of theoretical memoing [52]. All transcripts were coded by an experienced qualitative coder (T.W.) and 20% of transcripts were independently double-coded by the principal investigator (J.C.). A third team member (M.K.) resolved any disagreements through consensus with the two coders. Coding was completed using NVivo 10 (QSR International Pty Ltd., Version 8, 2008).

RESULTS

Quantitative results

Only one of eight therapists met our threshold of an average score of 80% (5.6/7) fidelity on each therapy session (one therapist did not enroll participants due to unforeseen circumstances). When the fidelity threshold was lowered to 57% (4/7) average score on each session, five therapists at two different sites were successful. If a median score of 5/7 on each session was set as the criterion, three therapists out of eight met this threshold. Mean fidelity scores and standard deviations for each item from Sessions 1–3 were compared with those reported by Forsberg et al. [42]; a study designed to evaluate the fidelity measure (see Table 1). The scores from the Forsberg study are quite similar to ours, suggesting that fidelity in our study was adequate.

Table 1|.

FBT fidelity scores

	N	Min.	Max.	Mean	SD	Forsberg et al. (2015) Mean	Forsberg et al. (2015) SD
Session 1 Greet family	8	3	7	4.62	1.41	4.47	1.27
Session 1 Take history	8	2	7	5.62	1.60	4.40	1.37
Session 1 Externalize	8	2	7	4.87	1.55	3.98	1.33
Session 1 Intense scene	8	3	7	4.75	1.58	4.03	1.56
Session 1 Charge parents	8	3	7	5.12	1.24	4.00	1.10
Session 2 Take history around food	8	3	7	5.12	1.24	4.42	.97
Session 2 One more bite	8	3	7	5.00	1.31	3.43	1.45
Session 2 Align pt with Siblings	7	3	7	5.14	1.46	3.27	1.70
Session 3 Focus discussion on food	8	3	7	5.25	1.39	4.45	1.02
Session 3 Help parental dyad with refeeding	8	3	7	5.37	1.30	3.85	1.02
Session 3 Siblings efforts	8	3	6	4.50	1.07	3.50	1.26
Session 3 Modify criticism	8	3	7	5.00	1.31	3.34	1.75
Session 3 Distinguish interests from AN	8	5	7	5.62	.74	4.12	1.23

FBT, family-based treatment; AN, Anorexia Nervosa.

No significant correlations were found between fidelity total score from Sessions 1 to 3 with baseline BIRCS, EBPAS, OLS, MPAQ, and ORC scores (see Table 2). We also examined correlations of total fidelity with age, number of years in current role, and decision-making capacity and found no significant associations (see Table 2). Paired t-test analysis showed that these variables did not change significantly from pre- to postimplementation (see Table 3).

Table 2|.

Correlations between total score on first three sessions of FBT and participant characteristics at baseline

		BIRCS	EBPAS	OLS	MPAQ	ORC	Age	Years in current role	Decision-making capacity
FBT fidelity	Pearson Sig N	−0.577 0.135 8	−0.469 0.241 8	−0.685 0.061 8	−0.607 0.110 8	−0.686 0.201 5	0.007 0.987 8	0.171 0.685 8	−0.091 0.829 8

ORC, organizational readiness for change scale; OLS, organizational learning survey; BIRC, brief individual readiness for change; EBPAS, evidence-based practice attitudes scale; FBT, family-based treatment.

Table 3|.

Paired t-tests for pre- and postimplementation intervention measures

Measure	Pre	Post	Difference	N	t	Df	Sig
ORC	419.3	411.7	7.7	6	1.674	5	.155
OLS	74.8	79.9	−5.1	14	−1.466	13	.166
MPAQ	61.9	64.1	−2.2	15	−1.167	14	.263
BIRC	11.7	12.5	−.8	17	−1.496	16	.154
EBPAS	39.1	41.1	−1.9	14	−1.393	13	.187

ORC, organizational readiness for change scale; OLS, organizational learning survey; BIRC, brief individual readiness for change; EBPAS, evidence-based practice attitudes scale.

Qualitative results

Overall experiences

Participants reported that our implementation approach differed from other models they had experienced in that the ongoing supervision was “built-in.” One participant said that “supervision is the key to fidelity” as it imparts a level of accountability. Another participant had this to say about our implementation approach:

Well the workshop was supremely helpful. You know to monitor and the opportunity to ask questions and to try to role play I think is key…..So most of the time you’re just trying to teach yourself from a book. And so any opportunity to practice it is key and then, you know, when you’re actually applying it, to have had those regular supervision phone calls was essential and being able to trouble-shoot and to know when you are off track and it made a difference for sure. (site 1, P3)

When asked about their overall impressions of the implementation approach, all participants indicated that they would continue to deliver FBT according to the manual, and they were continuing to include key elements such as; family meals, weighing patients, plotting weights on a graph, and involving siblings. For example, one participant stated the following:

The process of going through this study made it quite clear to me the pieces that were missing and how valuable those pieces are. Most notably, the family meal, sibling involvement, just essential pieces of manualized, formalized FBT that we had not, that I had not been doing before. (site 1, P5)

And another participant had this to say:

We thought we were doing FBT before, but we really weren’t. We weren’t doing the family meal. You know, we weren’t following the stages as recommended in the manual and I think we tightened up on that and that has made a difference. (site 4, P17)

Training workshop

All participants described having medical practitioners and administrators present at the training workshop as helpful. Participants enjoyed the role-play aspect the most, and asked for more of this practice. One medical practitioner said the following:

I think it was very important to be present to be able to academically understand the treatment modality and therefore be on the same page as the remainder of the team providing the treatment. This ensures fidelity to the model in my understanding from all disciplines. (site 3, P10)

Another participant explained that having the administrators there was very helpful in terms of buy-in:

…what I felt was most helpful for our team was getting the admin person on board…The admin person who may be more behind the scenes generally, to be included in all of this giving them more insight to something that they know we do, they support what we do but they don’t really know what it is. So that was helpful. (site 3, P12)

Clinical and implementation consultation

With respect to clinical consultation, participants reported that the recording and review of sessions increased their fidelity to the model. Some indicated that more frequent and more immediate feedback would have been helpful. One participant said the following:

mostly in knowing that this case was being recorded to somebody else on the other end listening every once in a while, so it just made me more conscientious in following the model. (site 2, P8)

One participant stated it helped with confidence-building:

I think the education and feedback that we received I think it was quite helpful. I really appreciated that. It gave me a sense of confidence too. You know, it relieved some anxiety if you weren’t sure what you were doing of if you were feeling stuck, so that was helpful. (site 4, P17)

With respect to implementation calls, participants described that having administrators on the calls was essential to ensure study success. One administrator had this to say:

it was really helpful for me to be able to get the information to you guys to know what needed to be done. Like you know uploading all the data, you know…but just the administration part of things that did keep the study going. (site 2, P9)

Fidelity rating

Fidelity rating provided by the external expert was thought to be helpful by all therapists, however, some therapists did not find self-rating to be helpful as it was hard to be objective, and many therapists did not find parent-rated fidelity helpful, as parents often provided very high scores. One participant commented as follows:

I don’t see myself as a valid rater. I’m biased, so I give myself a neutral…..I had one set of parents who gave me perfect all the way through, regardless of how badly or how well I did, and another set of parents who were a little bit more discerning…. (site 2, P8)

Another participant commented that the fidelity measure helped her to stay on track:

I think what the primary tool that I was really using were those fidelity measures to be honest. And I kind of felt like that if most of those things were happening then I was staying on track. (site 3, P13)

Ideas on how to improve and sustain fidelity

Participants made several suggestions for improving and sustaining FBT fidelity including a web-based forum that would provide clinical support for physicians and therapists providing FBT. All participants desired ongoing contact with FBT colleagues and experts to improve and maintain fidelity to the model. One therapist had this to say:

Like some type of an FBT listserv where you know we’re connected in some way. Like in terms of the supervision stuff right? So, it could be we get feedback in terms of the mentorship component and then somehow feeding a listserv into that would be helpful because it kind of keeps things going, keeps the momentum going . . . so having access to somebody, mentorship, when you know you have these tough cases and you’re stuck. Just for consultation right? And that’s were I go back to the listserv. (site 4, P17)

Another therapist said this:

If there’s a way to create some kind of community of online learning or professional development. So whether that’s for example could take the form of maybe an email list where different topics discussed or articles to present or updates, or training events…. (site 1, P6)

Many participants mentioned that linkage to an academic center and involvement in research were beneficial to their clinical programs. Some participants suggested that a website hosting training videos would be beneficial, and that the website could also share the latest research articles. One participant said this:

any sort of updates on studies or trials, any research that has come out recently on FBT, that would be kind of neat…. (site 4, P16)

Participants were keen to continue to enhance fidelity internally, within their own teams. One participant suggested an FBT checklist specifically for medical practitioners who are providing medical management would be helpful. Others suggested that peer fidelity rating should be implemented within their programs, whereas others did not feel comfortable with this concept and felt that external fidelity review would be more objective. Most participants mentioned they preferred group supervision so they can learn from each other, although a minority of participants preferred individual supervision. One participant thought their team should meet regularly to boost their fidelity to the model:

I’m wondering too, could our team create our own little booster session or refresher every year to get a better understanding of what’s accurate to the model, almost like a little retreat, you know? (site 1, P3)

Several participants identified that training new staff in FBT would be challenging and suggested developing province-wide training in FBT and family therapy, as this participant identified:

There should be a province-wide, I’m not sure, I want to use the word “get-together” but there should be a province-wide meeting for individuals that are doing family-based, whatever you want to call it, family therapy with eating disorder families on a regular basis. (site 2, P8)

DISCUSSION

This is the first study to use mixed methods to evaluate a blended implementation approach in the pediatric eating disorders field. Informed by the AIF, CFIR, and IO frameworks, our implementation approach consisted of: (a) preparatory site visits; (b) the establishment of implementation teams; (c) a training workshop; (d) monthly clinical consultation; (e) monthly implementation consultation; and (f) fidelity assessment. Implementation success was evaluated using FBT fidelity ratings.

Only one therapist out of eight was successful in attaining fidelity scores of 80% (5.6/7 or higher) on each session of FBT. When the threshold was lowered to an average fidelity score of 4/7 (57%) or above on each session, five therapists out of eight attained this. When a median of 5/7 on each session was used, three out of eight therapists met this cut point. These results lead us to consider which criterion could be a standard indicator of implementation success. A previous study examining fidelity ratings using this measure report mean scores in the 3–5 range on Sessions 1–3 in a highly trained and supervised group of 19 clinicians providing FBT in a randomized trial [42]. This study was not available at the time that we designed our current study, and thus could not be used as our predetermined benchmark. An additional dissemination study examining fidelity scores with this measure captured mean scores in the 4–6 range on the first three sessions of FBT among three clinicians [53]. Given these findings, perhaps our expectation of 80% fidelity on each session was too ambitious, as our fidelity scores are actually well aligned with previous studies.

Our decision to use a score of 80% fidelity on each session was based on expert opinion, as no papers had been published on this topic at the time. Prior research on family therapy fidelity in the field of psychosis indicates that 80% (5 out of 7) is a reasonable cut point in determining treatment fidelity [54], however, it is difficult to compare our fidelity instrument to the one used in the psychosis study. A recent review of fidelity assessment in CBT trials laments that adequate fidelity is not typically reported in psychotherapy trials and is not well-defined [55].

Interestingly, measures of readiness, attitude, and self-efficacy were not associated with fidelity and did not change from pre- to postimplementation. Lack of association could be due to a small sample size or insufficient variability in the data. However, other authors also have found similarly surprising results. Pemberton et al. [56] reported that therapist confidence, attitudes toward EBTs, and workload were not predictive of participation in training following a workshop for a trauma-focused CBT intervention for children and adolescents. However, perceived learning during the CBT workshop was a predictor of consultation call participation [56]. The construct of perceived learning may be a related concept to certain personality variables found to be important in another study examining the implementation of the Family Check-Up, an early intervention designed to reduce various child problem behaviors [57]. These Family Check-Up researchers found that an experienced, high readiness subgroup of health care providers demonstrated high levels of agreeableness, conscientiousness, openness, extraversion, as well as, positive attitudes toward evidence-based practice, high levels of work-related enthusiasm, and personal well-being [57]. The likelihood of being in this experienced, high readiness subgroup also was related to higher fidelity for Family Check-Up implementation [57]. We did not measure learning during our workshop, nor did we examine personality characteristics; these variables would be important to measure in future studies.

An additional implementation study by Barwick et al. [7] found minimal pre–post differences in terms of readiness, attitudes, and self-efficacy. This study utilized the AIF while looking at CFIR constructs related to the implementation of motivational interviewing (MI) at four sites treating children and adolescents with mental health difficulties [7]. These researchers noted a reduction in motivation for practice change and a reduction in individual readiness for change from pre- to postimplementation, and postulated that the reductions were due to a realization by clinicians of the complexities of practice change; beliefs that developed during the implementation study. These authors also found that fidelity to MI emerged over time and that fidelity measurement in practice proved to be challenging for the clinicians involved in the study. They note that some practitioners found audiotaping therapy sessions for fidelity assessment to be the most challenging aspect of the study [7], findings similar to our own.

Our qualitative data indicated that participants across sites found the implementation approach helpful, and all intended to continue providing FBT and striving for fidelity in their clinical practice. Most participants found ongoing supervision and fidelity rating by an external expert to be most important in our approach. Some implementation activities were found to be onerous, such as audio-recording and uploading therapy sessions to the website for expert review. While participants reported all elements of our implementation approach to be helpful, they also offered suggestions for additional supports, including; an annual workshop or booster session within their programs, regular supervision from external FBT expert, developing a community of practice within the province that would gather several times a year, and virtual options for continued contact with other FBT therapists and experts, including a listserv or website containing training videos and updates of the latest research.

Limitations to our study include its small sample size. We trained eight therapists to take on one to two FBT cases, setting the stage for a larger randomized study that will examine how fidelity relates to patient outcomes. Although fidelity ratings did not meet preestablished criteria, this likely reflects an overly stringent benchmark for fidelity, as our scores were similar or greater than those in a recent study examining the application of the fidelity measure with highly trained clinicians [42]. No data exist to indicate how many cases or how much time and support are needed for therapists to become competent in FBT; this could be a focus of a future study. Our exploratory study revealed a lack of association between baseline variables and fidelity. This could be a reflection of our limited sample size, or an issue of not capturing specific variables such as perceived learning during the workshop, or personality variables found to be important in other recent studies, such as agreeableness, conscientiousness, openness, and extraversion. It would be important to measure these variables in future studies. In addition, we are unable to determine the most clinically meaningful fidelity criterion, as we did not evaluate clinical outcomes. A future larger trial will evaluate these patient-level outcomes. Although the role of the medical practitioner in supporting the FBT model was not a focus of this study, this would be an important aspect of a future study, particularly as this role can be quite critical in the success or failure of FBT. Fidelity rating is time-intensive and laborious. Innovative strategies to efficiently measure fidelity should be a focus of future research, along with the development of a fidelity measure for the medical practitioner. In addition, we did not measure the cost or sustainability of our intervention. Continued efforts to sustain fidelity to the FBT model, such as the utility of a listserv or website as suggested in our qualitative findings, would also be important areas of future research. Despite limitations, our findings suggest that our blended implementation approach is feasible within Ontario’s ED network, and is highly acceptable and desirable to clinicians, medical practitioners, and administrators.

CONCLUSIONS

An implementation approach consisting of: (a) preparatory site visits; (b) the establishment of implementation teams; (c) a training workshop; (d) monthly clinical consultation; (e) monthly implementation consultation; and (f) fidelity assessment is feasible and valued by clinicians, medical practitioners and administrators working with children and adolescents with eating disorders. Further research is needed to determine how to operationalize adequate fidelity to the FBT model and what additional implementation strategies could be added to improve the implementation and sustainability of FBT in clinical practice.

Compliance with Ethical Standards

Conflict of Interest J.L. receives honoraria from the Training Institute for Child and Adolescent Eating Disorders, LLC, and royalties from Guilford Press and Oxford Press. All other authors have no conflicts to declare.

Author’s Contributions J.C. conceived the research idea with input from J.L., M.K., G.M., M.B., A.N., C.W., and S.F. J.C. was primarily responsible for the overall study design, overseeing the project, analysing the data, and drafting the manuscript. T.W. also analysed the qualitative data with input from M.K. All authors read and edited the manuscript, and approved the final manuscript.

Ethical Approval All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Ethics approval was obtained through the Hamilton Integrated Research Ethics Board and all participating sites. This article does not contain any studies with animals performed by any of the authors.

Informed Consent Informed consent was obtained from all individual participants included in the study.