Therapist and supervisor perspectives about two train-the-trainer implementation strategies in schools: A qualitative study

Abstract

Background

Train-the-trainer (TT) implementation strategies (in which designated clinicians are trained to then train others in an intervention) are promising approaches to support mental health clinician use of evidence-based interventions in school contexts. However, there is little evidence to date examining clinicians’ perceptions of the acceptability and feasibility of TT strategies, or comparing clinicians’ perceptions of different types of TT strategies.

Methods

The current study was conducted as part of a larger hybrid effectiveness-implementation trial, in which school-based therapists and supervisors received one of two different types of implementation support to implement cognitive behavioral therapy (CBT) groups for anxiety: TT (i.e., initial training for therapists and supervisors) or enhanced TT (TT+; i.e., initial training for therapists and supervisors, and ongoing external consultation for supervisors). We used applied thematic analysis to compare qualitative interview transcripts from 28 therapist interviews and 33 supervisor interviews from therapists and supervisors who received TT or TT+ support and report themes that were similar and different across the two groups.

Results

Most themes were similar across the TT and TT+ conditions: therapists and supervisors in both conditions perceived the group anxiety intervention as acceptable and viewed supervision as acceptable, helpful, and feasible. Therapists and supervisors in both conditions had mixed impressions of the contextual appropriateness of the group anxiety intervention, and some reported logistical challenges with weekly supervision. Some unique themes were identified among the TT+ condition, including supervisors experiencing professional growth, and therapists and supervisors perceiving supervision as critically important and enjoyable.

Conclusions

These results suggest that TT implementation support, using a model in which an internal supervisor receives initial training and then provides ongoing supervision, is acceptable and feasible to support a group CBT intervention in schools. The results also highlight additional benefits that therapists and supervisors perceived when supervisors received ongoing consultation.

Clinical Trial Registration Information

The clinical trial from which these data were derived was registered at ClinicalTrials.gov (https://clinicaltrials.gov/) prior to the time of first patient enrollment. The registration number is: NCT02651402.

Plain Language Summary

It is important to understand how to best support mental health clinicians in using evidence-based interventions in school contexts. Train-the-trainer (TT) strategies, in which designated clinicians receive training to then support others in using the intervention, may be promising, and it is important to determine the optimal TT model. However, little is known about how clinicians perceive the acceptability and feasibility of implementing evidence-based mental health interventions with different types of TT support, which can include training and/or ongoing consultation for internal trainers. To address this gap, we used a qualitative approach to examine and compare themes from semistructured interviews with therapists and supervisors participating in two different types of TT implementation support to deliver an evidence-based group CBT intervention for anxiety within schools. Therapists and supervisors receiving both kinds of support had mostly positive perceptions of the intervention and implementation support. They viewed the group anxiety intervention as acceptable and impactful, although they had mixed perceptions about its contextual appropriateness. They viewed supervision as acceptable, helpful, and generally feasible, although challenging at times. However, unique themes among the group who received the more intensive type of TT support, such as supervisors experiencing professional growth over time, suggest that there may be additional benefits to the more intensive support. Overall, these results suggest that TT strategies are an acceptable and feasible approach to support clinicians in delivering group CBT in schools. This study highlights the benefit of embedding qualitative interviews within randomized controlled trials to better understand clinician experiences and perspectives.

Introduction

Mental health concerns such as depression and anxiety are common among youth (Merikangas et al., 2010) and have increased sharply among children and adolescents during the COVID-19 pandemic (Racine et al., 2021). Although evidence-based practices (EBPs) for the treatment of youth mental health problems, such as group cognitive behavioral therapies (CBT) for anxiety, offer promise (Chorpita et al., 2011), the majority of youth with mental health concerns do not receive treatment of any kind (Costello et al., 2014). When youth do receive mental health services, they frequently do so in schools (Ali et al., 2019; Duong et al., 2021). School-based mental health services offer a number of advantages compared to services provided in other settings, including promoting access (Ali et al., 2019) and reducing stigma (Atkins et al., 2017). The provision of mental health services in urban schools may be particularly important for reducing disparities in treatment access and outcomes (Atkins et al., 2015; Cappella et al., 2008), as school-based mental health services minimize many access barriers faced by families in marginalized communities (Atkins et al., 2015).

However, clinicians in community and school settings often do not use EBPs, such as CBT for anxiety (Eiraldi et al., 2015; Garland et al., 2010). In particular, clinicians in underresourced urban schools face many barriers to implementing EBPs across contextual levels (Domitrovich et al., 2008; Eiraldi et al., 2015). At the individual level, one key barrier is limited preservice training: therapists in this setting typically have a master's level education, but often do not have specialized training in CBT or other EBPs (Creed et al., 2013; Forman & Barakat, 2011). Other potential individual-level barriers include therapists’ professional stress and burnout, attitudes regarding CBT, and self-efficacy regarding delivering CBT (Domitrovich et al., 2008). Additional barriers at the level of school-based teams include a lack of adequate time for team-related activities (Bambara et al., 2009), and frequent turnover of team members (Herlitz et al., 2020), which leads to the need to retrain staff (see Wolk et al., 2019). At the school level, barriers include practical issues such as challenges in identifying an acceptable time or space in which to deliver interventions (Langley et al., 2010), as well as broader concerns about alignment with the educational mission of schools (Atkins et al., 2017). Finally, at the macro level, financial considerations can impede the delivery of EBPs in schools and other community settings (Okamura et al., 2018; Stewart et al., 2016).

Implementation strategies such as training (Herschell et al., 2010) and ongoing supervision or consultation (Bearman et al., 2013; Nadeem et al., 2013) are likely needed to support school-based clinicians in implementing evidence-based mental health practices with fidelity. Train-the-trainer (TT) strategies, in which designated clinicians or organizations are trained to then train others in the EBP (Powell et al., 2015), are viewed as important and feasible in school settings (Lyon et al., 2019). In addition to using a TT model to provide initial training, this model can also be applied to support ongoing supervision. For example, external consultants can train or consult with internal (i.e., agency-employed) supervisors who in turn supervise therapists. This may have the potential to support the delivery of CBT in schools; specifically, training and ongoing supervision may address individual-level barriers such as lack of training, and consultation with agency-employed supervisors may help build internal capacity within agencies. However, these implementation strategies can also be time-consuming and expensive, and may therefore be challenging to deploy given macro level, structural, and financial barriers. This makes it important to determine which implementation strategies are necessary and sufficient to support clinician fidelity to EBPs and child outcomes (Eiraldi et al., 2016).

In addition to potentially variable effects on implementation fidelity and child outcomes, implementation strategies may also vary in the extent to which clinicians perceive them as acceptable, feasible, appropriate in their context, and useful (Motamedi et al., 2021). For instance, among a sample of therapists trained in EBPs as part of a system-wide EBP initiative, therapists perceived four educational implementation strategies—including having access to ongoing consultation or supervision and having an internal supervisor trained in the EBP—as essential for EBP delivery (Motamedi et al., 2021). However, there is little evidence to date comparing therapists’ perceptions of the acceptability, appropriateness, and feasibility of implementing EBPs with different forms of implementation support.

Using a qualitative approach to understand clinician perspectives about how they experience implementation strategies such as supervision or expert consultation, as well as the EBPs they are asked to implement, is important for several reasons. First, clinicians are likely to engage in and sustain participation in implementation strategies only when they perceive them as acceptable and beneficial (Lyon et al., 2013). Second, implementation strategies such as supervision or consultation may affect clinicians’ experiences in ways that are not captured by quantitative measures of fidelity, but are nonetheless important to inform optimal implementation supports (Herschell et al., 2014). Finally, clinician perspectives about their experiences with implementation may help contextualize results regarding fidelity or effectiveness, for example, by providing a more in-depth picture of how clinician experiences did or did not differ between types of implementation supports.

Current Study

The current study was conducted as part of a larger Hybrid Type II effectiveness-implementation study of group CBT for anxiety in urban schools (Eiraldi et al., 2016). In the larger study, school-based therapists and their supervisors were randomized to one of three conditions with different anxiety interventions and implementation strategies (Eiraldi et al., 2016). In order to focus on the comparison between implementation strategies, the current study compares therapists and supervisors who were assigned to implement an 8-session, manualized, group CBT program that has been adapted to be more feasible, engaging, and culturally appropriate (i.e., CBT Anxiety Treatment in Schools [CATS]; Khanna, 2016) with one of two forms of implementation support. In the standard TT condition, therapists and supervisors both received an initial training focused on CBT principles for anxiety, best practices for group management, the treatment manual, and (for supervisors) best practices for supervision. Additionally, in the standard TT condition, supervisors were expected to meet weekly with therapists for supervision during their delivery of the group; they received checklists to guide supervision sessions, and supervision was expected to include review and discussion of group session video recordings. In the enhanced TT (TT+) condition, therapists and supervisors received the same initial training and had the same expectations for ongoing supervision. Unlike in the standard TT condition, supervisors in the TT+ condition received a weekly consultation session with a member of the research team to support their ongoing supervision.

Quantitative results from the larger study revealed that child outcomes did not differ between these two conditions. Additionally, therapist fidelity was slightly higher in the TT+ condition, although fidelity was high regardless of whether the supervisor received ongoing consultation sessions (Eiraldi et al., under review). These results raise questions about why child and fidelity outcomes were generally similar across conditions. Furthermore, the quantitative methods used to assess these outcomes may not be sensitive to differences in therapists’ or supervisors’ experiences across conditions, such as the extent to which they perceived the support they received as acceptable, appropriate, or useful.

In this paper, we expand on the main study results by examining whether the perceived acceptability, appropriateness, and feasibility of the group intervention and supervision differed when supervisors did or did not receive additional ongoing consultation sessions. Specifically, we use a qualitative approach to address the question: how did supervisors and therapists who implemented CATS with TT+ support perceive the acceptability, appropriateness, and feasibility of implementing group anxiety treatments within their role, compared to supervisors and therapists who implemented CATS with TT support? We also examined the perceived acceptability and feasibility of the consultation itself. By using qualitative methods to compare the experiences of therapists and supervisors when supervisors did or did not receive ongoing consultation, we aim to explain the quantitative results showing generally similar child outcomes and fidelity across conditions (i.e., expansion; Palinkas et al., 2011) as well as identify any ways the experiences of therapists and supervisors differed depending on the type of support they received (i.e., complementarity; Palinkas et al., 2011).

Method

Procedures

All procedures were approved by the school district research board and the Institutional Review Board (IRB) of the Philadelphia Department of Public Health. Qualitative interview data were collected as part of a broader, cluster-randomized, hybrid effectiveness-implementation trial. In that trial, clinicians from nine mental health agencies provided services for students in grades 4–8 in 36 public and charter schools in large urban school districts in the Northeast United States. The district's student body is racially and ethnically diverse, and approximately 80% of district students live in households that are income-eligible for free or reduced-price meals. After they completed leading or supervising a group, therapists, and supervisors were invited to participate in an interview with a member of the research team to share their experience. Participants provided verbal informed consent as part of their participation in the broader study. Participants were compensated for their time delivering group sessions and participating in supervision or consultation in the broader study, but were not compensated for participating in interviews.

Participants

The current study uses interview data from 28 therapist interviews (11 from the group that received standard TT support and 17 from the TT+ condition) and 33 supervisor interviews (13 from the group that received standard TT support and 20 from the TT+ condition) from therapists and supervisors who participated in the relevant conditions of the broader study. Participants were selected by inviting all available therapists and supervisors who had participated in the broader study to complete an interview following their participation. Five supervisors and eight therapists declined participation or could not be reached (87% participation rate for supervisors and 78% participation rate for therapists). Participants were approached by email, phone, or in-person to schedule the interviews, which took place by telephone at a time convenient for the participant. In most cases, participants were in a private space at their workplace (i.e., a school) during the time of the interview. The sample of supervisors was 85% female; 54% of supervisors were Black and 42% were White; 4% were Hispanic/Latino. Nearly all (96%) supervisors’ highest education level was a Master's degree, and one supervisor had a Doctorate. The sample of therapists was 84% female; 48% of therapists were Black and 48% were White; 4% were Hispanic/Latino. Nearly all (96%) therapists had a master's degree, and one therapist's highest education level was a Bachelor's degree.

Measures

Semistructured Interview

The semistructured interview protocol (see Supplemental Materials, File A) was designed to explore the process, feasibility, acceptability, and appropriateness of the group treatment sessions and the type of support that therapists and supervisors received. The interview protocol began with an open-ended probe (i.e., “First, I’d like to hear your general thoughts or experiences about participating in CATS this year”). The protocol then asked questions about the therapist or supervisor's experience with each of the key components of the project: training, recruitment, running groups, supervision, and consultation (for supervisors in the CATS with TT+ condition who received consultation). Follow-up probes included questions to assess acceptability, appropriateness, and feasibility, such as, “Was the amount of support and feedback you received from your supervisor helpful for you to run the group?” and “Was the expectation to have consultation once per week feasible?” Members of the research team (e.g., female research assistants, JG, QR-M) conducted the interviews, with training and supervision from PhD-level clinical psychologist members of the research team GML, RE. Research team members who conducted interviews were part of the same research team that provided training and consultation, but the individual team members conducting interviews had not worked directly with the therapists or supervisors. Qualitative interviews were audio-recorded, transcribed, and deidentified, and deidentified transcripts were used for coding and analyses. Interviews lasted on average 13.5 (SD = 7.4) min.

Data Analysis

The interview transcripts were coded in multiple stages (Saldaña, 2013). After an initial review of transcripts, we developed a codebook with definitions, keywords, and example excerpts for a set of 13 codes. This process used an integrated inductive and deductive approach, where codes included a priori constructs of interest (e.g., barriers and supervision) as well as constructs that emerged as salient across the interview protocols that were reviewed (e.g., sustainment and engagement). After the initial codebook was developed, one master coder (PhD-level clinical psychologist) and two research assistant coders (bachelors’ or masters’ level research assistants) applied the initial coding scheme to four interviews. They discussed the coding decisions, including any disagreements, and iteratively refined the codebook (MacQueen et al., 1998). Once the coding team was able to come to a consensus on all excerpts, the coding scheme was applied to all interviews using NVivo software. Sixty-one interviews (therapists and supervisors in the CATS TT or CATS TT+ condition) were used in the current analyses.

Coding was conducted by the two female research assistant coders, each of whom was the primary coder for approximately 50% of the interviews. To avoid coder drift, approximately 20% of the interviews were coded by both research assistant coders as well as the master coder. These interviews were discussed at biweekly meetings throughout the coding process. Any disagreements were resolved by discussion; for these interviews, the “consensus” coding decisions were used as the final coding. Minor clarifications to the codebook were made to clarify definitions and decision rules based on these discussions.

The current analyses used excerpts coded as “Consultation” (“when the supervisor in the TT+ condition discusses information about the consultation session that occurred with a research staff member as part of the study procedures”), “Supervision” (“when the interviewee discusses supervision. This includes statements about supervision made by the therapist as well as by the supervisor”), and “Intervention/Curriculum” (“when the interviewee discusses the content of the intervention”). We focused analyses on these codes, and not the “Training” code, because our goal was to compare between two conditions that differed in terms of ongoing implementation support (i.e., whether or not internal supervisors received external consultation), but did not differ in terms of the initial training.

We used applied thematic analysis (Guest et al., 2012) to examine similarities and differences in themes regarding consultation, supervision, and intervention between the two conditions (CATS TT+ and CATS TT) through a multiple-stage process. A member of the research team first prepared a concept-by-text matrix to organize coded text into themes for each first-level code (i.e., consultation, supervision, and intervention), condition (i.e., TT and TT+), and respondent group (i.e., supervisors and therapist). The team member then drafted a series of second-stage analytic memos, which the master coder vetted (Saldaña, 2013). The second-stage memos included summaries and illustrative quotes of the themes that were observed in each condition and respondent group, as well as a comparison of themes that were similar or different between conditions. The analytic memos were refined through an iterative process of discussion and revision among the research team, in which we synthesized the data regarding the most salient themes that were similar or different between conditions. We also examined the extent to which themes were similar or different between supervisors and therapists and whether this differed by condition. Themes relevant to the research questions about acceptability, appropriateness, and feasibility are reported here.

Interview participants did not provide feedback on the transcripts or findings.

Results

Most themes were similar across supervisors and therapists from the TT and TT+ conditions, but some themes were different between conditions. In this section, we summarize findings related to perspectives about the intervention (i.e., CATS group therapy model) and supervision that were similar between conditions. Then, we summarize findings related to TT+ supervisors’ reported experiences of consultation—the component unique to this condition—and contrast with TT perspectives related to a desire for additional support. Across themes, we note when perspectives from supervisors differed from therapists’ perspectives and when the degree of consensus between the two respondent groups differed by condition. Finally, we provide a summary of the results of the comparative analysis that showed differences between TT and TT+ conditions as related to perspectives of the intervention and supervision.

Common Themes Between Conditions

Table 1 displays selected common themes and subthemes and representative quotes regarding the intervention and supervision from supervisors and therapists between the TT+ and TT conditions. We describe these common themes in this section.

Table 1.

Common Themes Between the TT+ and TT Conditions Related to the Intervention and Supervision

Common themes between conditions and exemplar quotes
Theme	Supervisor quote(s)	Therapist quote(s)
CATS group intervention generally perceived as acceptable and beneficial
The CATS intervention was perceived as broadly acceptable.	“I like the fact that it allowed me to…steer away from what we do every day, to be able to participate in this, specifically with the anxiety study, with the kids, to help support them because we see them every day and we’re doing different types of group, but this was more specialized.” (Supervisor in TT+ condition; S21022) “I think that what really drives this project is that the curriculum is really good.” (Supervisor in TT condition; S11007)	“I thought the groups were really fun.” (Therapist in TT+ condition; T32047) “So, I loved running the group. I think it's really great because it gave these kids an opportunity to […] see that they are not going through this alone. They are not the only one that feels anxious or worried about school or other things. So, I really liked that, and I just really liked running the group.” (Therapist in TT condition; T12019)
Therapists specifically liked the rewards system that was included in the CATS intervention.	“I think that the therapist liked the idea of the ticket system and of […] the rewards.” (Supervisor in TT+ condition; S41041) “The second part I liked about it was the reward system that was set up during the individual sessions for the children.” (Supervisor in TT condition; S41065)	“I liked the curriculum, I love the motivating toys, I love the ticket system, I like that the kids got into it.” (Therapist in TT+ condition; T42059) “I like that the tickets aren’t taken away, it's all about positive. […] and then they loved the prizes, like that was a motivation, I could kind of feel if they didn’t want to kind of participate, like they remembered, ‘Well I want the prizes, so….’” (Therapist in TT condition; T12019)
Therapists and supervisors believed that they observed a change in participating children as a result of the intervention.	“And I liked being able to see the transformation of a couple of our kids.” (Supervisor in TT+ condition; S11012) “I definitely think the kids that attended and were serious about the group got a lot out of it.” (Supervisor in TT condition; S31028)	“It really all came together, and it was very educational for me, and I think that the kids picked up on a lot of things, too, to help themselves with anxiety, and that was great.” (Therapist in TT+ condition; T42057) “It was just cool to see them like, again, using the videos, like you start to see them understanding it, and that is just really cool.” (Therapist in TT condition; T12019)
Supervision generally perceived as acceptable and helpful
Supervision is broadly acceptable and beneficial due to feedback, emotional support, problem-solving support, and motivation	“Just to offer the encouragement, the appraisal, let her [the therapist] see the actual, look at the video and say, ‘Look you are doing a great job.’ […] So, I think that the positive feedback and encouragement is what kept her going.” (Supervisor in TT+ condition; S21022) “I just think she just felt in general, the support of me looking at some of the positive things she had done in the group was helpful. Getting her to see things that maybe she had missed during the actual supervision because she was the one running it, so getting that, and also getting some suggestions to interventions both that could be for the next group or outside of the group, for individual sessions for the child to help foster a better, healthier sense of group.” (Supervisor in TT condition; S31028)	"I felt like it was very helpful because they gave me like different pointers, different things that I could do in different groups and like how to improve the groups so that was helpful with just watching and critiquing and seeing what we can do to improve the group for next week.” (Therapist in TT+ condition; T22030) “It was very positive. It thought it was a good use of our time, my time. […] I think we were able to solve the problems as well as get some feedback on how the session was going.” (Therapist in TT condition; T22024)
Structured nature of supervision as acceptable and helpful	“I like doing supervision anyway, so it was really nice doing it and having that structured curriculum to go off of and it kind of helped drive the supervision a little bit…” (Supervisor in TT+ condition; S31029) “As far as the checkoff lists […] I thought they were really helpful in guiding what I’m looking for in the videos and also how I can help guide as far as supervision with the clinician.” (Supervisor in TT condition; S11007)	“I think the supervision, the weekly supervision helped out a lot with me remembering where certain things were and then to help guide me more about the purpose and the goal of the group.” (Therapist in TT+ condition; T42061) “…we would each have our manuals out, which was always helpful, and kind of going through what would happen the next week, we would do that together…” (Therapist in TT condition; T12019)
Weekly supervision generally perceived as feasible, although challenging at times, due to logistical barriers, billing expectations, and time constraints
Weekly supervision is perceived as generally feasible but sometimes challenging, due to logistical barriers, billing expectations, and time constraints	“Yeah, it was okay. It was a little struggle… at times but it was okay. We, we were able to get through them. But a lot of times, there was conflict with scheduling and with different, like, say school would be closed on certain days so then it would run over into the next week. But it wasn’t too bad though.” (Supervisor in TT+ condition; S22030) “Time was a challenge… just depending on whatever was happening at the school… So the challenge was getting the supervision done so we could do the group.” (Supervisor in TT+ condition; S21029) “So, it was kind of difficult for the clinician and I to meet at times we had scheduled to meet. So, I would have to cancel often, like sometimes two and three times in a row. So, and by that time, she had started another session without us even getting supervision out for the previous session. Sometimes we had to like double up.” (Supervisor in TT condition; S21026)	“Yeah [it was feasible], we had a couple weeks where we struggled but, for the most part we were able to manage it.” (Therapist in TT+ condition; T12011) “It was kind of like we pretty much did it but sometimes it might not get done on the day that we wanted to because things might come up. But overall, I think we did a pretty good job of staying on pace. You just have to sometimes kind of close the door and put up a sign and not answer the door.” (Therapist in TT condition; T42060)
Perceived appropriateness of the intervention varied, and therapists and supervisors sometimes made adaptations for their context
Therapists and supervisors had mixed perspectives about the appropriateness of the intervention for the students in their group; some had concerns that the intervention was “too content heavy” or that the reading level was too advanced.	“It seems… like, the content, there's a lot of content to squeeze in in a short of amount of time. So, I think that put pressure on the therapist…” (Supervisor in TT+ condition; S21011) “I had to kind of help her with fidelity at times because we found that the curriculum was at a higher level than our children, and so it had to be broken down.” (Supervisor in TT condition; S21015)	“Appropriate age group, academic levels, and things like that, I would have different […] structured trainings for different academic levels if that makes sense. Some of our children struggle and have IEPs so I think if there's a child that struggles with reading, and I learned that through my actual group, this go around, that it could be a little–it could present as a challenge for that child, especially because most of content requires them to have to read and do the assignments in the book, so that could be a struggle for somebody that struggles with some academic reading….” (Therapist in TT+ condition; T32033) “I had a hard time for some of those really meaty sessions with the amount of content that I was supposed to cover in those sessions. It was a lot especially because, you know, the kids that we’re working with in the school, there's a lot of behaviors to manage. So, I feel like maybe if I didn’t have to deal with behaviors, maybe the content I could have gotten through, but there were some weeks where I feel like I didn’t even get through half of the content and then I felt like I wasn’t set up for the next week.” (Therapist in TT condition; T22020)
Therapists experienced challenges engaging students during at least some sessions.	“Some weeks the children were more engaged and actually following along with the material, other weeks there was a lot of distractions. I don’t know if they were less interested in the material that was presented at that time, but it was a lot of distractions, like kids not paying attention to what we were talking about and just being disruptive and not following along, not staying seated, and then bothering the camera too.” (Supervisor in TT+ condition; S22030) “It was difficult to try and keep the fidelity sometimes, because the feedback that we were getting from the kids, from the group, was like, “Oh this is boring,” or you could tell that they were kind of zoning out, but we had to keep the fidelity.” (Supervisor in TT condition; S11018)	“However, there was also a disadvantage with sticking to the script because I think for the kids, it became a little bit too tedious. And they didn’t like the fact that […] we had to go along with this whole curriculum and I think that they would have liked it more if we were able to improvise a little bit more and to have a little bit more fun with it, but due to time constraints, we weren’t able to really do much with it except for stick to the script which seemed to be long and tedious at times, and by the fourth group, the kids were over it.” (Therapist in TT+ condition; T31043) “I think by the end it got a little repetitive with [one component of the intervention sessions]. Like they were like, ‘We have to do another one!?’” (Therapist in TT condition; T22027)
Therapists reported making small adaptations to the curriculum, including shifting examples to make them more relevant to the students in the group and adding more hands-on activities.	“No, we definitely used what was provided with the manual. Only thing we might have did was add a little bit of twist to some of the activities. Like, we might have them get up or we might add like a couple prompts […] like just to have them moving around more. I think you have to have the kids move around to keep them interested. So just more so just us incorporating different things that make them more interested in the content.” (Supervisor in TT+ condition; S31043) The curriculum “had to be broken down, […] so we had to figure out how to do that, and so a lot of our time was spent on me trying to figure out what the compromise was going to be because our kids struggled with in the way it was supposed to be delivered.” (Supervisor in TT condition; S21015)	“I did use examples that specifically pertained to the children rather than always use the examples in the book.” (Therapist in TT+ condition; T32043 “I tried to follow the manual for the most part… Examples maybe I changed [to examples] that were maybe more relevant to them, but other than that I stuck to the content.” (Therapist in TT condition; T32046)

Note. CATS = cognitive behavioral therapy anxiety treatment in schools; TT = train-the-trainer; TT+ = enhanced train-the-trainer.

CATS Group Intervention is Generally Perceived as Acceptable for Therapists and Supervisors and Beneficial for Children

Across both conditions, therapists and supervisors described the CATS intervention as broadly acceptable. They described enjoying the content, finding the groups “fun” to lead, and appreciating that the groups gave them an opportunity to provide students with more specialized support regarding anxiety. Several therapists specifically liked the rewards system that was included in the intervention; they described this as helpful, enjoyable, and motivating for the students in the group. In addition, similar in both conditions, several therapists and supervisors shared that they liked that they perceived the group intervention as having an impact on the students in their group. For example, one therapist in the TT condition said, “You start to see them understanding it and that is just really cool (T12019).”

Perceived Appropriateness of the Intervention Varied and Some Respondents had Concerns About the Pacing of the Intervention or Students’ Engagement in it

Similar across both conditions, therapists and supervisors varied in their perceptions of the appropriateness of the CATS intervention for the students in their group. A few participants noted that they believed the curriculum was at an appropriate level and was accessible to the students in their groups. Other participants shared that they thought the content of the CATS intervention was too challenging for the students in their group (e.g., in terms of reading level or pacing). Several respondents from both groups thought that some sessions were too content heavy. One TT therapist explained,

I think maybe two of them [group sessions] I felt like okay this is a lot of stuff, will I be able to get through it? And we were able to get through it. You know, I did have to maybe…pick and choose here and there on those two sessions (T32046).

This theme was similar across conditions, but within the TT group specifically, this theme was more salient among therapists than supervisors. Relatedly, therapists and supervisors in both conditions reported experiencing challenges engaging students during at least some sessions. For example, one supervisor in the TT+ condition said, “Some of the material was a little bit lengthy in terms of the kids paying attention, and it's kind of a lot to get through in a session…” (S31027).

Therapists and Supervisors Sometimes Made Adaptations for Their Context

Relatedly, several therapists and supervisors discussed adaptations that they made or supported the therapist in making. For example, one supervisor in the TT condition discussed encouraging the therapist she supervised to add opportunities for movement during sessions:

So I told her, like, for one of the activities we’re doing to just [give the students] the ability to get up and maybe do this, or get up and write something on one of the sticky boards on the wall, or something of that sort (S41065).

Therapists in both conditions also shared that they personalized examples to make them more relevant. For example, one therapist in the TT+ condition described, “I would definitely say I tried to tailor the subject manner to what the kids are going through, like things they’re doing in the school” (T32038).

Supervision Was Generally Perceived as Acceptable and Helpful

Similar across both conditions, supervisors and therapists generally perceived supervision as acceptable and beneficial to therapists due to the feedback, emotional support, problem-solving support, and motivation provided. Several supervisors and therapists in both conditions shared that they liked that supervision provided an opportunity for positive feedback and emotional support. Others reported that they used supervision to problem-solve challenges and prepare for upcoming sessions. Both therapists and supervisors between conditions described this as being generally acceptable and motivating. In both conditions, several supervisors reported that they appreciated the structured nature of supervision and described how structured materials (e.g., manuals and checklists) strengthened supervision.

Weekly Supervision Generally Perceived as Feasible, Although Challenging at Times due to Logistical Barriers, Billing Expectations, and Time Constraints

Finally, in both conditions, supervisors and therapists shared that they perceived the expectation of weekly supervision meetings as generally feasible, although some described this as challenging at times. For instance, some supervisors and therapists shared challenges carving out time for supervision due to competing demands or scheduling conflicts. For example, a TT supervisor said,

My schedule is jammed pack, so it was kind of difficult for the clinician and I to meet at times we had scheduled to meet. So, I would have to cancel often, like sometimes two and three times in a row (S21026).

Another TT supervisor mentioned challenges with billing expectations: “The agency wanted us to, you know, just bill a certain amount of hours and we are like, ‘We have to do these supervisions, it's like part of this group’” (S41057). Despite reported challenges, respondents generally described the expectation of weekly supervision as ultimately feasible.

Themes Unique to the TT+ Condition

In the TT+ condition, supervisors received an additional implementation support: ongoing consultation. Here, we provide an overview of findings from the applied thematic analysis that showed several themes related to the perception of the intervention and supervision that emerged in the TT+ condition, but not the TT condition. In general, therapists and supervisors in the TT+ condition had more positive attitudes toward the intervention and supervision.

Supervision Perceived Overall as Critically Important and/or Highly Enjoyable

Although therapists and supervisors in both groups reported generally positive views about supervision as a source of feedback and support, a few supervisors and therapists in the TT+ condition expressed specific views that supervision overall was critically important or enjoyable. For example, one supervisor shared: “I think the most important part is the supervisions. I’m not sure about any other groups that do not have the supervision. But I think that is a vital piece to the whole group in itself” (S31043). Additionally, one therapist explicitly attributed the benefit they derived from supervision to the ongoing consultation received by their supervisor:

…it was helpful that she also had her own consultation with [the consultant], so I think that helped her and I think in turn that helped me. I feel like it would’ve probably been very difficult to make good use of that supervision if we had not gotten the additional consultation with [the consultant] each week (T32047).

Reviewing Video Recordings of the Group Therapy Session During Supervision as Clearly Acceptable and Specifically Highly Useful for Shaping Implementation

Many supervisors and therapists in the TT+ condition shared that they found it highly acceptable and useful to review video recordings of the group therapy session during supervision. Although a few supervisors and therapists in the TT condition described the videos as “helpful,” supervisors and therapists in the TT+ condition discussed in more detail the ways in which they used the videos in supervision and found them useful. For instance, one therapist explained, “They were helpful because she [the supervisor] would point out and highlight specific moments in the group with things she would show me what I did well and stuff like that” (T42061). Similarly, a TT+ supervisor who called using video recordings “the strongest thing I got out of this whole project” described reviewing videos as helpful for the therapist she supervised:

I just felt it was a very unique way to kind of be able to sit and, especially because we’re able to kind of review video things that we were doing, we were able to kind of challenge maybe some of our more difficult beliefs about ourselves as therapists and kinda review, like, “These are things you weren’t doing well, but I want you to watch how these students are doing while you’re engaging in this.” I feel like my clinician responded really well to kind of having that type of evidence to back up feedback to her… (S41052).

Supervisors Experiencing Professional Growth or Improvement Over Time

Finally, several supervisors in the TT+ condition shared beliefs that they had improved at supervision over time or experienced professional growth as a supervisor as a result of participating in the project. They attributed this to a combination of consultation, experience, and the expectations of participation (e.g., structured and in-depth supervision). Although one TT supervisor made a general comment about their growth as a supervisor as evidenced by their improved feedback (“it helped me be better with critiquing” [S41048]), several TT+ supervisors shared more detailed reflections about how their supervision improved over time. For instance, one TT+ supervisor shared,

I noticed at the beginning, like our first supervision [or] our first two was like hit and miss, or like, “Am I doing this correctly?” but then towards the end I find […] my supervisee as well as myself grew very confident […] (S31035).

The Intervention Materials Were Perceived as Easy to Use and Helpful

Compared to supervisors and therapists in the TT condition, supervisors and therapists in the TT+ condition described the intervention materials as helpful and easy to use. TT+ participants described specifically liking that the program is structured, “packed up for us,” and prepared ahead of time, and distinguished this from their previous experiences running groups without structured materials. For example, a supervisor explained, “I like the fact that everything is prepared ahead of time so the clinician doesn’t have to scramble to try to put together topics for each group, and everything was just laid out there for the clinician” (S31029). This theme was not salient among participants in the TT condition.

Perceptions of Consultation in the TT+ Condition

Supervisors in the TT+ condition reported positive feelings about their experience in consultation overall, perceiving consultation as acceptable, important, and helpful for several types of reasons (see the summary in Table 2 for a description of subthemes and representative quotes). First, several supervisors shared that they appreciated that consultation provided them with feedback to improve their supervisory practice. Second, they described consultation as helping them structure or prepare for supervision. Several supervisors also noted that they found the positive feedback in consultation to be motivating or encouraging. Finally, some supervisors also described consultation as useful in generating ideas or strengthening skills in working directly with children.

Table 2.

Themes Related to Consultation From Supervisors in the Enhanced Train-the-Trainer (TT+) Condition

Theme	Exemplar supervisor quote(s)
Consultation perceived as highly acceptable overall and reported positive feelings about their experiences in consultation in general	“They were good, the feedback was really good. With someone actually, like I said, overseeing and watching and saying you’re doing the right thing or some of the things that you can be doing differently. So, it's nice to always get someone else's perspective…and input, ways that we can sort of kind of get through the process of the supervisions like things I can add in, things I can do differently. So that was just very helpful.” (S21022)
Consultation provided supervisors with feedback (e.g., suggestions, recommendations, and insight) to improve their supervisory practice	[Consultation helped to] “identify some things I could’ve done differently in the supervision sessions.” (S31041) “I found them helpful because there were sometimes, things that maybe like I missed or we didn’t think of, or, at times when it just felt like there were a couple sessions like where like we just felt like, ‘What are we doing?’, it was nice to have like a completely outside person's opinion.” (S31034)
Supervisors perceived that consultation helped them structure or prepare for supervision	[Consultation] “really helped me to be more prepared for supervision, have more structure and really focus on how I was supporting the clinician.” (S31041)
Supervisors found it motivating and encouraging when the consultant provided positive reinforcement	[Consultation was a “confidence builder because she was able to point out some of the strengths and just ways that I could improve….” (S31046)
Supervisors perceived consultation as helpful in terms of generating ideas or building skills for working directly with the children	“She was also good in being creative on ways that we was trying to when we was trying stuff on, different techniques to support the kids’ behavior. She was supportive in that…” (S41043)

Desire for Additional Support in the TT Condition

In contrast, some TT respondents shared that they would have liked to have received additional support from content experts. Specifically, one supervisor reported the desire for occasional feedback about supervision (“Are we doing this the right way?” [S11018]) and another reported that the therapist needed more help with the materials. Additionally, one therapist in the TT condition reported that “it probably would have been helpful for me to be meeting with someone and getting feedback from someone who was more familiar with the program” (T22020). Two supervisors in the TT condition reported issues with technology, specifically viewing videos of group sessions for supervision. This theme was not present among supervisors in the TT+ condition.

Discussion

The goal of this study was to examine therapist and supervisor perceptions of their experiences delivering a school-based group intervention across two different forms of TT support. We used a qualitative comparative approach to examine how the perceived acceptability, appropriateness, and feasibility of the intervention and supervision differed across implementation strategy conditions. We also examined how supervisors who received ongoing external consultation perceived the acceptability and feasibility of that support. In doing so, we expanded on prior work about TT implementation strategies by providing an in-depth examination of therapist and clinician experiences across different types of TT support.

Overall, the results suggest that therapists and supervisors in both conditions perceived the intervention and supervision as generally acceptable and impactful. In general, these findings are consistent with the quantitative results of the larger study (Eiraldi et al., under review). Specifically, the quantitative results revealed that child outcomes did not differ between conditions and therapist fidelity was high regardless of whether the supervision received ongoing consultation sessions (Eiraldi et al., under review). Consistent with these findings, the current qualitative results indicate therapists and supervisors found that group intervention and supervision acceptable, feasible, and beneficial, regardless of whether they received TT or TT+ support, which suggests that the standard TT support (which included an expectation for agency-employed supervisors to provide weekly supervision) was adequate to achieve fidelity. These results are consistent with prior evidence about the importance of ongoing supervision (Bearman et al., 2013; Nadeem et al., 2013) and expand on prior evidence by highlighting the specific benefits of supervision perceived by participants, such as emotional support and problem-solving support.

Across both conditions, therapists and supervisors had mixed perspectives about the contextual appropriateness of the intervention for the students in their group. Therapists reported making small adaptations to the curriculum to improve its appropriateness for the context, such as making examples more relevant to the students in the group and adding more hands-on activities. In the larger trial, we found that therapists in both the TT and TT+ conditions had high levels of content and process fidelity (Eiraldi et al., under review). These results highlight the importance of supporting therapists in making adaptations that improve appropriateness without impeding fidelity or effectiveness.

Supervisors and therapists in both conditions generally reported perceiving the expectation of weekly supervision as feasible. However, some therapists and supervisors in both conditions discussed important challenges to weekly supervision. In particular, some participants described barriers such as scheduling conflicts, competing responsibilities, billing expectations, and difficulty finding a time and space to complete supervision without interruptions. Taken together, these results suggest that implementation strategies involving weekly supervision provided by an agency-employed supervisor (with or without consultation support), are generally feasible, but do not address key logistical and practical barriers to the provision of evidence-based mental health services in schools.

However, these findings also identified key differences between therapists’ and supervisors’ experiences across the two conditions of implementation support. Specifically, participants in the TT+ condition, compared to those in the TT condition, specifically stated that supervision was critically important and/or highly enjoyable, described reviewing videos of the group therapy session during supervision as highly acceptable and useful, and perceived the intervention materials as easy to use and helpful. It is possible that the ongoing, external consultation helped supervisors in the TT+ condition better understand how to effectively use video recordings in supervision and how to use intervention materials, which also led to more positive experiences for the therapists they supervised. Although supervisors in both conditions were expected to review videotapes and provide feedback during supervision, the current results suggest that this happened in a more consistent and meaningful way when supervisors received external consultation. These results highlight the importance of multifaceted implementation strategies, such as ongoing consultation support for the delivery of audit and feedback (i.e., internal supervisors reviewing videotapes of group sessions and providing feedback) implementation strategies.

We note several limitations. First, group interventions during the final year of data collection were halted prematurely when schools abruptly closed for in-person learning due to the COVID-19 pandemic; interviews were collected from therapists and supervisors during that year if they had completed at least four group sessions, but the fact that many of the groups were terminated early due to the pandemic may have affected their perspectives. Second, the individuals who provided consultation to supervisors in the TT+ condition and conducted the interviews analyzed here were members of the same research team. The individual members of the research team who provided consultation did not participate in conducting interviews; nevertheless, this structure may have affected participants’ willingness to be candid regarding aspects of the intervention or implementation support that they found less acceptable. Finally, therapists and supervisors in the current sample may have varied in their experience with CBT and conducting group interventions; these factors may have influenced their perception of the intervention, supervision, or consultation in important ways, but were not measured or examined here.

The current results have important implications for the development and selection of implementation strategies to support evidence-based mental health intervention in school settings. TT implementation strategies are widely used to support mental health service delivery, but it is important to identify the optimal type of TT support. These results suggest that TT implementation support, using a model in which an internal supervisor receives a one-time training and then provides ongoing supervision, is acceptable and feasible to support an evidence-based mental health intervention in an urban school setting. Given findings that this implementation strategy resulted in similar therapist fidelity and child clinical outcomes as the more intensive TT+ strategy at a lower cost (Eiraldi et al., under review), these results suggest that this type of TT support is a promising implementation strategy, and may be sufficient in many situations.

However, the results also highlighted additional benefits that therapists and supervisors perceived when supervisors received ongoing expert consultation. Supervisors found this consultation enjoyable, helpful, and important for their professional growth. Supervisors in this group reported experiencing professional growth over time, and therapists described supervision as critically important and highly enjoyable. Moreover, some clinicians who did not receive ongoing consultation expressed a desire for additional support. These differences in therapist and clinician experiences between the conditions were not captured by quantitative measures of fidelity and child outcomes, but may be important for clinician's ability to sustain implementation of the EBP or for reducing clinician burnout or turnover, which are pressing issues for mental health service delivery in community and school-based settings (Brabson et al., 2020; Eiraldi et al., 2015; Wiltsey Stirman et al., 2012). Examining these possibilities is an important direction for future research.