National Trainers’ Perspectives on Challenges to Implementation of an Empirically-Supported Mental Health Treatment

Abstract

This study examined perceived challenges to implementation of an empirically supported mental health treatment for youth (Trauma-Focused Cognitive Behavioral Therapy; TF-CBT) and explored the potential use of technology-based resources in treatment delivery. Thematic interviews were conducted with 19 approved national TF-CBT trainers to assess their perspectives about challenges to implementation of TF-CBT and to explore their perceptions about the potential value of innovative, technology-based solutions to enhance provider fidelity and improve quality of care. These data offer some important insights and implications for training in evidence-based treatments, provider fidelity and competence, and patient engagement, particularly for those interventions targeting trauma-related symptoms among youth.

One in four US children experiences a mental health disorder with severe impairment and/or distress during their childhood (Costello, Egger, & Angold, 2005; Kessler, Avenoli, & Costello, 2011; Merikangas, He, & Burstein, 2010). Ensuring that these children have access to the highest quality mental health care is a top public health priority. Efficacious child and adolescent treatments exist for a wide range of mental health disorders (Chorpita, Daleiden, & Ebesutani, 2011). However, these treatments are rarely delivered with satisfactory fidelity in community-based mental health service settings (Drake, Torrey, & McHugo, 2003; Garland, Brookman-Frazee, & Hurlburt, 2010; Kazak, Hoagwood, & Weisz, 2010; McHugo, Drake, Whitley et al., 2007; Raghavan, Inoue, Ettner, Hamilton, & Landsverk, 2010). This is not just a quality shortfall, but a quality chasm (McCabe, 2004), as an Institute of Medicine report concluded in a review of health care and mental health studies, stating that “there are large gaps between the care people should receive and the care they do receive” (Institute of Medicine; IOM, 2001, p. 236). Statewide and national dissemination and implementation initiatives are underway to narrow these gaps (Ebert, Amaya-Jackson, Markiewicz, Kisiel, & Fairbank, 2011; Karlin, Ruzek, Chard et al.; 2010; Saunders, 2009). However, the problem of provider fidelity continues to present major challenges that are critical to address if care is to be improved.

Fidelity to evidence-based treatments (EBTs) is variable, even among well-trained providers (Drake et al., 2003; McHugo et al., 2007; McCabe, 2004; IOM, 2001; Saunders, 2009). This is a major public health concern because high EBT fidelity is associated with better patient outcomes (Bond, Becker, & Drake, 2011; McHugo, Drake, Teague, & Zie, 1999; Schoenwald, Sheidow & Letourneau, 2004). Achieving a greater understanding of the challenges experienced by providers when implementing empirically supported mental health treatments, as well as the potential value of innovative solutions, is therefore a top priority. To this end, we conducted a qualitative study with 19 trainers who successfully completed a training program in Trauma-Focused Cognitive Behavioral Therapy (TF-CBT; Cohen, Mannarino & Deblinger, 2006), led by the treatment developers. TF-CBT is an evidence-based treatment that addresses symptoms of posttraumatic stress, depression and disruptive behavior among children exposed to traumatic events. TF-CBT is an ideal treatment model for a qualitative study of this nature because it is among the most well-established treatments for children in the mental health field, and because it uniquely addresses multiple symptom domains (e.g., posttraumatic stress, depressed mood, disruptive behavior). In this study, we assessed approved national trainers’ perspectives about challenges that providers experience when implementing TF-CBT, and explored their perceptions about the potential value of innovative, technology-based solutions to enhance provider fidelity and improve quality of care.

TF-CBT is a short term treatment model, comprised of eight specific treatment components that comprise the acronym “PPRACTICE” and includes Psychoeducation and Parenting skills; Relaxation; Affective expression and modulation; Cognitive coping; development and processing of a Trauma narrative; In vivo exposure; Conjoint sessions where the child shares their Trauma Narrative (TN) with a supportive caregiver; and strategies to Enhance future safety and development. TF-CBT has been evaluated in numerous randomized controlled trials and is among the most well-established and widely disseminated mental health treatments for children (Cary & McMillan, 2012; Cohen, Deblinger, Mannarino, & Steer, 2004; Cohen, Mannarino, & Ivengar; 2011; Deblinger, Mannarino, Cohen, Runyon, & Steer, 2011; Deblinger, Mannarino, Cohen, & Steer, 2006).

Method

Participants

Individual thematic telephone interviews were conducted with a sample of 19 mental health professionals who successfully completed a 15 month TF-CBT Train-the-Trainer program led by the TF-CBT developers (Drs. Cohen, Deblinger, and Mannarino). Because of their extensive training activities, all trainers worked closely with front-line providers and had first-hand knowledge of challenges to delivery of TF-CBT with fidelity. They were therefore ideally positioned to provide valuable qualitative data about common barriers and challenges faced by community-based providers, as well as to provide insight into the potential value of technology-based resources designed to improve quality of care. Collectively, the 19 professional trainers in our sample had been TF-CBT trainers for at least 3 years (M = 5.8, SD = 1.8), had at least 8 years of experience treating children (M = 17.1, SD = 5.2), and trained over 5,000 providers during the past year alone (M = 293.7, SD =190.7). Most of the trainers (89%) engaged in multi-component training activities that included in-person workshops and monthly or bi-monthly consultation calls with participating providers. Trainers were from all regions of the U.S.; most were women (83%) and non-Hispanic White (81%). The sample was evenly distributed between master's- and doctoral-level degrees; all were licensed mental health professionals.

Interview

The telephone interview consisted of two major sections: a brief section on interviewee demographics and experience (approximately 5 min), followed by a thematic interview assessing trainers’ perceptions about challenges experienced by providers when delivering TF-CBT (approximately 30 min). The thematic interview was semi-structured and addressed several issues relevant to implementation of TF-CBT with fidelity. First, trainers were asked specifically about TF-CBT components that they believed were most challenging for providers to deliver with a high degree of fidelity and competence. Follow up probes asked the trainers for their perceptions about why these specific components were difficult for providers. Second, trainers were asked which TF-CBT components they thought were the most difficult to engage children or caregivers. Follow-up probes inquired about the trainers’ perceptions on why these components were challenging for engagement. After these issues were addressed, trainers were asked an additional series of questions about what they perceived as the potential values and roadblocks associated with using technology-based resources to enhance provider fidelity and competence and patient engagement. This was described to them as follows:

One of the goals of our project is to develop technology-based tools to help providers deliver a high quality of care and keep families actively engaged in TF-CBT components, particularly those that are known to be the most challenging to deliver. The tools would be available on a tablet such as an iPad, and would include things such as interactive educational games, a trauma narrative writing tool, and video demonstrations to show caregivers specific behavior management skills.

Follow-up probes inquired about what features or topics should be included or excluded to ensure that these resources were engaging to children and their caregivers while also being useful to providers. Additional probes were used to generate ideas to ameliorate potential roadblocks to use of technology-based resources and to assess how their use might affect the therapeutic relationship. A copy of the full thematic interview is included in the Appendix.

Procedure

After IRB approval was obtained, we sent a single e-mail invitation to the full group of approved TF-CBT national trainers (n=51). We aimed to conduct between 15-20 interviews to achieve saturation (Morse, 2000). Participants were not offered compensation. Thirty trainers (59%) responded to our e-mail invitation and agreed to be contacted for interview. We were successful in scheduling interviews with 25 of these trainers, resulting in 21 completed interviews. Two interviews were not able to be coded for analysis due to audio-recorder malfunction, resulting in a total of 19 coded interviews. A postdoctoral fellow with training in, and experience delivering, the TF-CBT model administered the interviews. Interviews were 20-55 min (M=33) and were audio-recorded. Interviews were transcribed, coded, and checked for accuracy.

Results

Data Analyses

Given the limited research on trainers’ impressions of provider fidelity and competence, the qualitative approach chosen for this analysis is the constructivist grounded theory comparative method to code incident-by-incident (Charmaz, 2006). This approach was well suited for these data, as it takes the context of the question into account, acknowledges coders’ prior knowledge of research related to fidelity and competence and the influence of this experience on the coding processes, and provides guidelines for building conceptual frameworks between coded constructs. A doctoral level psychologist conducted primary line-by-line thematic coding using NVivo-9 (2010) to mark identified thick descriptors. Secondary and focused coding were then conducted to impose superordinate thematic codes and hierarchical structure through the methods of constant comparison and analytic induction (Glaser & Strauss, 1967). This process ensures both consistency in the coding and theme labeling within categories and divergence between coding categories. These results yielded 326 thematic codes. Consistent with Miles and Huberman's (1994) guidelines for similar sample sizes, only codes that held for at least three transcripts were retained, resulting in 249 themes. Themes were then organized into 17 superordinate thematic codes based on their presence, co-occurrence, and non-overlap in the transcripts.

The superordinate thematic codes (presented in Tables 1-4) fell into 4 larger domains: Fidelity, Competence, Engagement Barriers, and Opinions about the Technology-based Toolkit. During this stage of coding, Fidelity (Domain 1) was defined as provider problems in use of specific TF-CBT components; Competence (Domain 2) was defined as provider problems in knowing how to deliver the TF-CBT components well; Engagement barriers (Domain 3) were defined as problems getting patients to start, complete, or participate consistently in treatment and were divided further into predominantly caregiver vs. child barriers. For each of these 3 domains, specific TF-CBT model component superordinate themes were coded based on which of the TF-CBT model components (e.g., Psychoeducation, Relaxation) were indicated. Opinions about the Technology-based toolkit (Domain 4), defined as provider comments about the proposed toolkit, was further divided into anticipated barriers or problems with use of the toolkit, general benefits of the toolkit, and/or problems the toolkit may solve.

Table 1.

Domain 1: Themes Related to Provider Fidelity

Themes	# of Trainers Raising Theme	# of References to Theme Across Sample
Generally provider fidelity to trauma narrative sessions is poor	13	30
Generally fidelity to cognitive coping sessions is poor	12	26
Caregivers aren't given rationale in psychoeducation	6	6
Providers don't know how to involve caregivers in behavioral management	5	11
Providers generally have fidelity problems in teaching parenting skills	4	8
Providers fail to cover safety planning and body safety	3	4
Providers don't address caregivers’ cognitive distortions about trauma	3	3

Table 4.

Domain 4: Themes Related to Opinions about e-resources for TF-CBT

Superordinate Categories	Sub-Themes	# of Trainers Raising Theme	# of References to Theme Across Sample
Benefits of eTF-CBT toolkit	General provider excitement and enthusiasm	13	38
	Toolkit will improve child-provider relationship	11	14
	Toolkit will enhance child engagement and/or persistence in treatment tasks	9	33
	More tools/activities will improve competence	9	11
	Toolkit will improve provider fidelity to treatment model	9	16
	Toolkit will enhance child's perception of treatment (fun)	5	7
	Toolkit is portable and can be transported (outreach)	3	4
	Toolkit will enhance provider buy-in and investment in therapeutic model	4	4
	Tech toolkit will not cause any harm	4	6
Barriers to eTF-CBT toolkit use	Many aren't good with technology and object to it	8	12
	Don't use an iPad, no one has iPad	7	14
	Toolkit will not be TF-CBT and providers will mistakenly believe it is	7	17
	Technology will become an additional barriers to therapy	7	9
	IPad are too expensive for regular therapists	6	9
	Many agencies providers lack access to even computers	6	8
	Providers won't feel comfortable using if personal information is entered	6	6
	Agencies won't fund iPad	5	12
	Privacy becomes a problem if any personal info is entered into application or web	5	8
	Providers will use toolkit as babysitter for child patients instead of for therapy	5	6
	Tech can't be used in outreach due to lack of internet or devices	5	5
	Technology will harm privacy of narrative	4	5
	Therapy rooms lack web access, often purposefully	4	4
	Competency in rapport is already too low, toolkit will further hurt provider-patient relationship	4	4
	Older therapists won't use these tools	3	4
	Additional web security will be necessary if narrative included	3	3
eTF-CBT product will be usable and helpful because of excellence of other similar tools	Development group has previously developed useful and helpful tools (e.g., TF-CBTWeb and CBTWeb)	4	5

Inter-rater reliability

Approximately 20% of interviews were randomly selected and coded by a second, doctoral level, coder to assess overall inter-rater reliability. As recommended by Bakeman and Gottman (1997), discrepancies in coding between the primary and secondary coder were resolved through discussion until consensus was reached. These assessments were compared using NVivo-9 to calculate transcript level kappa coefficients and then computing average kappa's for each theme. Kappas above 0.60 are rated as reliable (Pelligrini, 2004) and indicate consistency in coding occurrence, commission, and omission, adjusted for chance. Of the 249 themes that resulted from the secondary coding, 219 had good-to-excellent levels of inter-rater reliability and were retained (κ range: 0.61-1.00, κ M = 0.95). Thus, only 30 themes were unreliable; these were not included in discussion. Kappa values for the 17 retained superordinate thematic codes ranged from 0.61-1.0 (M=0.88), also indicating good-to-high levels of inter-rater agreement. The resulting reliably coded themes, number of trainers that addressed each theme, and the number of references to the theme across the sample are presented in Tables 1 through 4.

Verification

The resulting data interpretations also were reviewed for validity through member checks (Kvale, 1996; Manning, 1997). In this case, several TF-CBT trainers, the interviewer who conducted the thematic interviews, and the TF-CBT developers reviewed the identified themes and gave feedback as to their validity. Our data interpretations were reported to be generally consistent with their impressions.

Domain 1: Provider Fidelity

As seen in Table 1, most of the trainers believed that there were some problems with whether or not providers implemented specific TF-CBT components, which we categorized as ‘fidelity’ to the model. These concerns fell into several TF-CBT components specifically related to trauma narrative development (13 trainers), cognitive coping (12 trainers), psychoeducation (6 trainers), parent behavior management (5 trainers), and enhancing safety (3 trainers). In identifying enhancing safety as a component that poses a challenge to fidelity, one trainer commented, “Well, it is not a difficult component, but it is that providers forget it like you drop it off the end of the model and, you know it comes right after the narrative session which a lot of providers have a really big build up to.” One explanation for not implementing the trauma narrative was related to provider avoidance. As one trainer noted, “Yeah, I mean the trauma narrative is difficult because I think there's a lot of avoidance on behalf of the clinicians. They're concerned that they're being mean. They're concerned that the child's not ready....”

Domain 2: Provider Competence

Most of the trainers believed there were significant problems in providers’ understanding of and abilities to deliver EBTs more generally, and TF-CBT more specifically. We categorized these responses as problems in ‘competence.’ As one trainer explained, “I think there are a lot of clinicians who have a very nondirective approach to therapy and I think they really struggle with aspects of TF-CBT because of that.” Another trainer noted, “I think that is what gets in the way of cognitive processing, too. They do not have enough training themselves in how to do basic CBT behavioral analyses and [cognitive] processing.” Several TF-CBT components identified by the trainers as containing skills that providers either had difficulty doing well or did not fully understand are common to a variety of cognitive-behavioral EBTs. For example, trainers had concerns about providers’ skills in using cognitive therapy generally (14 trainers), exposure sessions (12 trainers), challenging negative thoughts in cognitive processing (10), addressing (8 trainers) and appropriately responding to avoidance (8 trainers), and flexibly adapting the model (7 trainers) to the specific patient.

Regarding TF-CBT specifically, trainers identified development and processing of the trauma narrative (10 trainers) and psychoeducation (5 trainers) as components that posed challenges to provider competence. Trainers’ concerns about psychoeducation were related to providers talking at the child rather than with them about the trauma (5 trainers). For the trauma narrative, trainers were concerned about providers’ abilities to: determine the necessary level of detail (6 trainers), respond appropriately to child emotions (7 trainers), deal with their own fear and avoidance (4 trainers), direct the cognitive processing of the narrative (6 trainers), and facilitate sharing of the narrative with the caregiver (5 trainers). A number of trainers noted that challenging negative thoughts during cognitive processing of the narrative was particularly difficult (10 trainers). One trainer stated,

I think that these skill sets are probably the least trained skill sets in clinical programs and so I think that the therapist themselves just have the least comfort in applying these skills...You know it's one thing to kind of try to do cognitive processing and it's another thing I think to be skilled and adept at it. I think that they just haven't had good solid training in that.

Similarly, another trainer stated,

In terms of the cognitive processing component, I think that's a skill set that's the most new in terms of the skill set for most trainees that I've been working with...it's harder for them to sort of learn and wrap their brain around what the goal is and how it's done most successfully.

Domain 3: Client Engagement

Although many trainers noted that child engagement was generally not a concern (6 trainers), others suggested that avoidance posed challenges with some of the model components (Table 3). Of particular note, trainers expressed their belief that trauma avoidance is a significant issue in child engagement (7 trainers) and that avoidance of trauma cues at the onset results in children disengaging before treatment even really begins (4 trainers). One trainer explained, “There are very reserved children who are mostly avoidant. That can be difficult at all the phases of the model because the child is scared or resistant or not engaged in the therapy process overall.”

Table 3.

Domain 3: Themes Related to Child and Caregiver Engagement

Superordinate Categories	Sub-Theme	# of Trainers Raising Theme	#of References to Theme Across Sample
Child engagement	Avoidance symptoms are the cause of child engagement problems	8	15
	Trauma avoidance is a significant problem	7	14
	Engagement of child patients isn't problematic for any components of this treatment protocol	6	14
	Engagement of child patients in trauma narrative sessions isn't problematic	4	10
	Avoidance of trauma cues at start of treatment and in narrative process leads to disengagement	4	5
Caregiver engagement	Caregivers don't buy into TF-CBT treatment process and rationale	7	9
	Caregiver symptomatology and mental health limits engagement	6	29
	Caregiver avoidance of hearing about trauma results in caregivers disengagement	6	10
	Caregivers believe treatment shouldn't involve caregivers, only children	3	3

Whereas avoidance also was an issue for caregiver engagement (6 trainers), other issues emerged. Specifically, several trainers noted that caregivers’ own mental health problems can negatively affect engagement in treatment. One trainer noted that caregivers’ mental health problems require that providers put in considerable extra work, stating that

...a lot of times the parents are so low functioning that they are just dealing with their own trauma. Dealing with case management and handling these things up front- if you don't handle them, then it can become a big barrier down the line to doing the rest of treatment; they actually at some point undermine what the therapist is trying to do with psychoeducation for the child.

Trainers also highlighted caregivers’ inabilities to buy into treatment as a major barrier to engagement (7 trainers). One trainer noted that many caregivers believe treatment should only involve the child (3 trainers), making it difficult to convey that “helping them (the caregiver) understand that in this treatment there is a parallel parent component and that the degree to which you can help your kid get better is depending on their participation.”

Domain 4: Opinions about the Proposed e-TFCBT Toolkit

Finally, we inquired about trainers’ beliefs regarding the utility of an e-based toolkit (e.g., IPad/tablet) to facilitate TF-CBT implementation and whether they perceived any barriers to its use (see Table 4). Overall, trainers shared positive opinions about the idea of a tablet (13 trainers), stating that it had the potential to improve the child-provider relationship (11 trainers), as well as positively impact fidelity (9 trainers), competence (9 trainers), and engagement (9 trainers). As one trainer explained,

...I think the less the clinicians have to worry about the mechanics of what they are doing, and find adequate resources to do so, I think that allows them to relax a little bit more and pay attention to the engagement and potential challenges or threats to engagement.

Another trainer noted,

I think kids would feel very engaged. The child's generation connects through technology, right? I mean, they actually connect much less face to face. So, I think that it's embracing that generation. I think it makes therapy possibly more accessible to a kid who is used to connecting with friends through texting and Facebook.

Beyond provider buy-in, trainers also believe this tool can enhance children's beliefs in the effectiveness of TF-CBT (4 trainers). Trainers also indicated that this tool might have tremendous utility for treatment delivered in non-traditional, community-based settings (3 trainers). As one trainer highlighted,

...A lot of clinicians (who) go to homes or, you know do outpatient care in the community at homes or schools. Then I could see it being a huge benefit there if they could take it with them so that they do not have to bring their big filing box with all of their different themes and activities....

In contrast, many trainers noted that an e-toolkit could be a barrier to therapy. For example, one trainer stated,

I think it is marvelous that we have lots of tools and resources at our disposal to help playfully and interestingly engage our clients, and to see how people are interested in things like electronic media and apps for phones and things that are attractive and novel...but it can actually end up serving the role of reinforcing avoidance ... my biggest concern that these things don't get to be seen as the end in themselves and they are simply a means to an end.

Additionally several providers expressed doubt regarding the usefulness of a tablet-based toolkit for providers, many of whom they believe are uncomfortable with technology (8 trainers). One trainer stated, “I think there's always going to be those therapists who are less comfortable with online technology but I see that becoming a smaller and smaller problem.” Several trainers noted that if personal information could be entered into the application, this would lead to privacy problems (5 trainers), would become problematic for the privacy of the trauma narrative (4 trainers), and that they would be unwilling to use the application (6 trainers).

Some trainers also were concerned that providers may rely too heavily on a tablet-based toolkit when delivering TF-CBT instead of using it as a supplemental tool (5 trainers) and that this might negatively affect the provider-patient relationship. Specifically, trainers worried that if rapport and competency were already low, a toolkit could further hurt the provider-patient relationship (4 trainers) and that some providers might use the tablet computer/iPad as a ‘babysitter’ for child patients (5 providers). Similarly, many trainers were concerned that providers would think that using the toolkit alone would be sufficient to call the treatment ‘TF-CBT,’ leading to providers deviating from the protocol and having poor fidelity to the model (7 trainers). As a way to address this problem, one trainer highlighted the importance of developing a toolkit so that “it somehow communicates that this is an enhancer and is not a replacer.”

Finally, trainers noted that providers may lack access to the equipment necessary to use an e-toolkit, such as an iPad/tablet (7 trainers), desktop computer (6 trainers), or internet access (6 trainers). One trainer stated, “Okay, well the biggest challenge I know even for our agency would be that we don't have computers in our therapy room.” Trainers expressed concern about the cost of the devices and the required internet, stating that regular therapists could not afford these devices (6 trainers), that patients would not have access to them or to the internet to use them in their homes (5 trainers), and that agencies would be unwilling to provide them due to their high cost (5 trainers).

Discussion

The primary aims of this study were to identify the most common challenges experienced by providers in the implementation of a trauma-focused evidence-based mental health treatment. TF-CBT (Cohen et al., 2006) was selected as an exemplar because of its extensive empirical support as well as our recognition that it shares common core components with other evidence-based, cognitive-behavioral mental health interventions for youth. In the present study, thematic interviews were conducted with a sample of key stakeholders, approved TF-CBT national trainers. These trainers have extensive involvement in training front-line mental health providers as well as significant direct experience in delivery of the model, which positioned them well to provide valuable data regarding common challenges experienced by providers in treatment delivery.

Trainer Perspectives on Challenges to Provider Fidelity, Competence and Child/Caregiver Engagement

Consistent with previous research (e.g., Allen & Johnson, 2012), many of the trainers reported that gradual exposure and cognitive coping presented significant challenges to treatment fidelity and provider competence. Explanations included provider discomfort in directive treatment approaches, as well as limited skill in delivering treatment and engaging patients in these components. Despite extensive data indicating the effectiveness of these treatment components (Bisson, 2007; Carr, 2004; Deblinger et al., 2011; Silverman, Pina, & Viswesvaran, 2008; Silverman, Ortiz et al., 2008; Wethington et al., 2008), many providers express discomfort and fear that these components will cause harm and undue distress. Concerns that such discomfort and fear can affect the fidelity and quality of care were reinforced by findings of the current study. Innovative solutions are clearly needed to identify new techniques that can assist providers in overcoming these barriers—for example, via specialized training and practice. Research exploring new directions in this area may have far-reaching implications.

A second focus of the interviews was to examine challenges to engagement. Although many trainers did not believe child engagement was problematic, those that did frequently identified patient avoidance as the specific barrier to engagement with both children and adults. Thus, as a means of enhancing engagement and reducing the likelihood that avoidance can negatively affect treatment, it is important to address these issues directly at the outset and throughout the treatment process and to provide a strong rationale for all of the treatment components (Cohen, Berliner & Mannarino, 2010; Cohen et al., 2006). It is also important to ensure that providers are appropriately trained to recognize and creatively address subtle forms of avoidance that may be driving disengagement. Due to the multi-component, complex structures of many child mental health treatments, these types of critical skills are often underemphasized in training, and deficits in these skills often go unrecognized in practice.

Trainers also believed caregivers’ own avoidance interfered with their abilities to engage in treatment. Other specific challenges to caregiver engagement included caregivers’ insufficient beliefs in the treatment process/rationale, their own symptoms or mental health problems and/or their beliefs that their child should be the sole focus of therapy. Whereas research has emphasized the importance of caregiver involvement to facilitate positive clinical outcomes (Dowell & Ogles, 2010; Deblinger, et al., 2006; Deblinger, Lippman, & Steer, 1996), data from the current study indicate that engaging the caregiver can be especially challenging. Thus, it is important that providers develop the skills to recognize situations where a caregiver's own mental health symptoms interfere with engagement in treatment, and to make referrals for individual treatment when indicated. Providers need to highlight the integral role of the caregiver in the therapeutic process from the outset of treatment and explicitly and repeatedly provide the rationale for the treatment approach. These issues are relevant to many therapeutic interventions for children because of the integral role of the caregiver in the child's recovery (AACAP, 2010; Dowell & Ogles, 2010; McKay, Pennington, Lynn, & McCadam, 2001). Finally, it is important to highlight that empirically supported strategies have been identified to address logistical and perceptual barriers to initial family engagement (McKay & Bannon, 2004; McKay, Lynn & Bannon, 2005) and to enhance active participation throughout all phases of the treatment process (Tuerk, McCart, & Henggeler, 2012). These should be incorporated as key components of child and family-based treatments.

Trainer Perspectives on the Potential Value of Technology-Based Resources

Many of the trainers noted that a tablet-based toolkit could assist providers in overcoming some of these identified challenges to the child-provider relationship, treatment fidelity, provider competence, and child and caregiver engagement. For example, a notable threat to provider fidelity is lack of engagement with patients at various phases of treatment. The use of innovative, technology-based learning activities in session offers a possible way to keep patients engaged and to reduce potential boredom and disinterest in the treatment exercises, particularly when working with children and adolescents (Matthews, Doherty, Sharry, & Fitzpatrick, 2008). Use of an iPad, computer, or other mobile or web-based device has the potential to increase engagement, particularly given children's growing familiarity and comfort in an increasingly technology-driven world (Rainie, 2009). Youth often have ready and instant access to technology in all phases of their lives, making it likely that this would enhance their interest, investment, and engagement in the therapy process.

Regarding provider fidelity, inclusion of material within a tablet-based resource that directly maps on to the specific model components may help keep providers on track, facilitate progress through the treatment model, provide reminders of treatment goals, and generate creative activities and ways to supplement the key principles/techniques being taught in therapy. These resources also have the potential to reduce therapist drift by providing tangible reminders of key treatment components.

Whereas many trainers positively endorsed the use of technology, there were also reservations. Predominantly these related to concerns that agency-based or private-practice providers have poor access to the internet and/or computers (tablet or otherwise) either in their office or while engaged in community-based outreach services, limiting their ability to use this type of resource. Trainers also expressed apprehension about cost as well as general openness and skill in using technology, making it possible that these resources would only be accessible to a certain segment of the population and thereby limit widespread dissemination and reach. Research that develops and evaluates these types of technology-based resources should prioritize cost-benefit analyses to ensure that sufficient data are available to agencies to assist them in estimating the value of such investments to their patient population and in terms of their ability to offer high-quality care.

Questions regarding confidentiality, web security, and general privacy also were raised as potential drawbacks to the use of an e-based toolkit. Thus, any e-based toolkit must include significant safeguards to protect privacy and confidentiality. And finally, trainers indicated that fidelity may be compromised if providers mistakenly believe that sole reliance on a toolkit is equated with successful completion of a complex, multi-component treatment model, such as TF-CBT. This highlights the importance of educating providers on the judicious use of a technology-based resource as a way to enhance and supplement, rather than detract from, the therapy process.

Limitations

This study had several limitations. First, thematic interviews were conducted with a small sample of professionals engaged in a specific evidence-based treatment protocol. Whereas the core TF-CBT components (psychoeducation, parenting skills, relaxation strategies, cognitive coping) are common to many other EBTs used in youth mental health, this is still a specialized treatment approach. It is possible that these findings may have relevance to other child-oriented CBTs, but it is premature to draw this conclusion without additional research.

Second, these data were provided by trainers regarding their perceptions of provider behavior. In most cases, trainers were relying on information drawn from provider self-report and questions raised during consultation calls or workshops, discussions with other trainers, or behavioral rehearsal activities during training workshops. It is likely that a minority of the trainers had first-hand knowledge of provider behavior via direct observation of treatment sessions with actual patients. However, this was not directly assessed in the current study, and the extent to which trainers’ perceptions were based on direct observation of provider behavior is unclear. Third, few inferences can be drawn about trainers’ perceptions of providers’ strengths in delivery of TF-CBT because the emphasis of the interview was on problems or challenges to implementation.

A final limitation concerns the definitions of fidelity and competence used in the current study. As noted in the literature (Fairburn & Cooper, 2011; Perepletchikova, Treat, & Kazdin, 2007; Schoenwald, et al., 2011), there are considerable challenges to the reliable and valid measurement of these constructs, and, in the present study, no specific guidelines or definitions were provided to interviewees to insure that they were differentiating between these constructs. However, inter-rater reliabilities were in the good-to-excellent range (κ range = 0.61-1.00, κ M = 0.95) indicating that interviewees and coders were able to make these distinctions. Further, there appeared to be differences regarding respondents’ opinions on the types of challenges faced by providers in choosing to use a model component (i.e., fidelity) vs. delivering the model with competence. It is likely that greater focus on therapist skill results in greater model adherence, and this continues to be an important focus for future research. But, perhaps the most important issue is to determine the ‘tipping point,’ namely at what point is a treatment delivered ‘well enough’ to achieve a desired clinical outcome? Fairburn and Cooper (2011) refer to this as ‘therapy quality’ and highlight that this may be more critical than definitional distinctions, particularly in clinical practice settings. We concur that this is an important and fruitful area for future research.

Conclusions

These data offer some important insights and implications for training in EBTs, provider fidelity and competence, and patient engagement. Trainers were generally consistent in their beliefs that certain CBT components pose more challenges to providers than others. Thus, it may be important for trainings to provide more in-depth focus on these key components – for example, specific content and opportunities for behavioral rehearsal in gradual exposure and cognitive processing components appear to be integral to helping providers achieve and maintain fidelity to this type of treatment approach. Further, it appears that trainers should not overlook the importance of ensuring that providers have basic CBT skills, rather than making a priori assumptions about the skill level of training participants. This speaks to the importance of pre-training assessments as well as engagement and discussion with agency leaders, supervisors, and the providers themselves to determine the level of provider skill prior to training and to incorporate basic therapy skill training if this is indicated.

Third, recent technological advances offer an opportunity to improve quality of care significantly by supporting the effective delivery of best-practice interventions. Such resources should be easy to use and should support the therapeutic alliance to allow efficient and effective integration into everyday practice. A technology-aided approach, if effective, may have potential to influence the way that evidence-based treatments are delivered in community mental health settings. To this end, initial exploratory research studies are needed that direct the process of developing such resources and that examine their potential to improve the quality of mental health care throughout the course of treatment. Findings from the present study highlight trainers’ perceptions that technology may be a valuable resource to facilitate treatment delivery if used properly, and that such resources may have potential to enhance provider fidelity and competence and increase patient engagement.

Table 2.

Domain 2: Themes Related to Provider Competence

Themes	# of Trainers Raising Theme	# of References to Theme Across Sample
Providers lack competency in cognitive therapy skill areas	14	59
Generally providers lack competency in exposure sessions	12	30
Providers lack competency in challenging negative thoughts in cognitive processing	10	33
Providers are unable to identify cognitive distortions	11	24
Lack competency noticing and addressing patient avoidance	8	15
Providers don't understand how to do cognitive coping sessions	8	15
Lack necessary skills in flexibly adapting TF-CBT	7	22
Lack competency responding to child's emotions appropriately	7	14
Lack competency in flexibly adjusting/adapting EBT to patient	7	14
Providers lack competence in cognitive processing of trauma narrative	7	13
Lack competency assessing level of detail necessary for narrative	6	22
Providers don't understand trauma narrative development process	6	15
Providers lack flexibility in adjusting cognitive coping to case and trauma	6	9
Providers lack competency sharing narrative with caregiver	5	6
Providers lack flexibility in adapting/teaching psychoeducation	5	5
Providers don't know how to creatively apply cognitive coping activities with children	5	5
Providers talk at children during psychoeducation and don't teach	5	5
Lack competency in exposure-based therapies	4	5
Providers fear and avoid hearing child's story	4	4
Providers do not competently transition from narrative and cognitive coping	4	4
Providers lack competency in sharing narrative	4	4
Providers can't manage patient crises and progressing through therapy	3	5
Providers lack competency in assessment	3	6
Lack competency in cognitive therapy	3	4
Providers don't get adequate training in model before being given complex cases	3	4
Providers lack competency in explaining treatment rationale to child	3	4
Competence in cognitive coping skills is not problematic	3	4
Providers don't use Socratic questioning in psychoeducation	3	3

Appendix

Semi-Structured Interview of Approved TF-CBT National Trainers

Thank you for participating in our survey. The survey should take about 30 to 45 minutes. First, we will ask you questions about who you are and your experience as a TF-CBT Train-the-Trainer. Next, we will tell you about our plans to develop a technology-based resource designed to help providers deliver TF-CBT with high fidelity and competence and to promote child/caregiver engagement in treatment. We will then ask you a series of questions to get some ideas from you about how we can make this tool most helpful for providers in their work with abused or traumatized children and their families. This interview will be audiotaped so that we don't miss anything. Do you have any questions?

Section A. About You and Where You Work

A01. OK, let's start. These first questions will address your experience providing treatment to children and adolescents. First, how long have you been conducting therapy with children?

A02. How long have you been conducting therapy with children who have experienced a traumatic event (such as sexual abuse, physical abuse, witnessed violence, natural or man-made disaster)? (months or years)

A03. Please estimate how many abused/traumatized children you have treated in the past year. If UNSURE, give ranges: (a) 1-5, (b) 6-10, (c) 11-20, (d) more than 20. If number is high and/or seems like a large part of caseload, what percentage of your caseload would you estimate is comprised of abused/traumatized children?

A04. What is your primary role? (Supervisor, Provider, Administrator, Other [specify])

A05. Next, a few more questions about your background for statistical purposes. What is your discipline? Are you a clinical psychologist, social worker, counselor, physician, or something else [specify]?

A06. What degrees do you hold? (MD, PhD, MSW, EdD, LMSW, LISW, PsyD, Other)

A07. Are you licensed? (Yes, No)

A08. [Interviewer circle gender—do not ask: male, female]

A09. Would you mind telling me your age? [If REFUSE, move on]

A10. Would you mind telling me what ethnicity/race do you identify with? [If REFUSE, move on]

A11. What state do you live in?

A12. In what year did you complete the TTT training?

A13. What type of TF-CBT specific training activities do you provide? (check all that apply: in-person 1-day workshop; 2-day initial/beginner workshop; 2 day advanced/booster workshop; telephone consultation calls; other – specify)

A14. Approximately how many TF-CBT workshops have you provided in the past year?

A15. Approximately how many providers have you trained in TF-CBT over the past year?

Section B. Orienting Participants to Project Goals and Website Content

[Orient participants to the broad purpose of our project using the two items below.]

[Briefly cover these issues (30-60 seconds)]

B1. The goal of our project is to explore the usefulness of technology-based resources for TFCBT to enhance provider fidelity to the treatment model and increase child/caregiver interest and engagement

B2. These resources would be used by providers in session to assist in engaging children and their caregivers, and to help them stay on protocol

Section C. Semi-Structured Interview

C1. What TF-CBT components do you believe are the most challenging for providers to deliver with high degree of fidelity and competence? (provide list of components for TF-CBT-PRACTICE)

[If participant only provides vague descriptions of TF-CBT, prompt them to tell you what specific PRACTICE component(s) appear to present the most challenge for providers]

C1a. [For each component selected] In your opinion, why do you think these components present the most challenge for providers?

[If participant needs prompting provide these examples: provider discomfort; provider lack of familiarity with procedures; lack of congruence between the treatment model and the provider's theoretical orientation or prior experience; limited experience with traumatized youth, or youth more broadly]

C1b. [For each component NOT selected] So, to verify, it is your opinion that providers generally do not have too much difficulty administering the following TF-CBT components with fidelity and competence?

[If answer is no, seek clarity on other components that present challenges and why]

C2. Keeping children engaged in treatment is a common obstacle to completing treatment and staying on protocol. In your experience, which TF-CBT components are most challenging in terms of actively engaging the child? Which components are most challenging in terms of engaging the caregiver? (Provide list of components for TF-CBT PRACTICE)

[If participant only provides vague descriptions of TF-CBT, prompt them to tell you what specific PRACTICE component(s) appear to present the most challenge for client engagement]

C2a. [For each component selected] In your opinion, why do you think these components present the most challenge for client engagement?

[If participant needs prompting provide these examples: client discomfort/avoidance; boredom; lack of perceived relevance; provider enthusiasm; provider competence]

C2b. [For each component NOT selected] So, to verify, it is your opinion that providers generally do not have too much difficulty engaging children and/or caregivers in the following TF-CBT components with fidelity and competence?

[If answer is no, seek clarity on other components that present challenges and why]

C3. One of the goals of our project is to develop technology-based tools to help providers deliver a high quality of care and keep families actively engaged in TF-CBT components, particularly those that are known to be the most challenging to deliver. The tools would be available on a tablet such as an iPad, and would include things such as interactive educational games, a trauma narrative writing tool, and video demonstrations to show caregivers specific behavior management skills. In your opinion, how can we make sure that this toolkit adequately addresses the challenges that providers experience in delivering TF-CBT?

[Follow-up prompts:]

C3a. What features would be most important for this toolkit to be engaging to children?

C3b. Caregivers?

C3c. If the toolkit is going to be useful, providers will need to find it valuable. How can we make this valuable to them in their practice?

C4. Do you think that a technology-based toolkit has potential to enhance provider fidelity or competence?

C4a. Why or why not? [will it reduce providers’ discomfort with delivery of certain treatment components? Which ones?]

C4b. How do you think it could affect the therapeutic relationship? Is it likely to help it? Hurt it? Not affect it at all?

C4c. Do you think the toolkit would be useful in addressing challenges to successfully completing TF-CBT?

C5. What types of concerns or challenges do you think are important to consider in developing this web-based resource to facilitate delivery of TF-CBT?

[Follow-up prompt]:

C5a. Do you think there is a potential for harm? [if YES] Why?

C5b. IF YES: Can you think of some solutions to these challenges? How can we address these issues that you mentioned?

[Follow-up prompt]:

C5c. What are some key barriers to using the resource that providers will have? (aside from cost/accessibility)

C5d. Web-security concerns?

C5e. Privacy concerns?

C5f. IF YES TO ANY OF ABOVE: Can you think of some solutions to these challenges? How can we address these issues that you mentioned?

D. Completion of Interview

That completes the questions I had for you. We appreciate your time and thoughts. If you have questions or comments, please feel free to call. Do you have any questions before we wrap up?