The Juggling Act of Supervision in Community Mental Health: Implications for Supporting Evidence-based Treatment

Abstract

Supervisors are an underutilized resource for supporting evidence-based treatments (EBT) in community mental health. Little is known about how EBT-trained supervisors use supervision time. Primary aims were to describe supervision (e.g., modality, frequency), examine functions of individual supervision, and examine factors associated with time allocation to supervision functions. Results from 56 supervisors and 207 clinicians from 25 organizations indicate high prevalence of individual supervision, often alongside group and informal supervision. Individual supervision serves a wide range of functions, with substantial variation at the supervisor-level. Implementation climate was the strongest predictor of time allocation to clinical and EBT most relevant functions.

Introduction

A substantial number of evidence-based treatments (EBTs) for child and adolescent mental health problems exist, many with multiple randomized controlled trials (RCTs) supporting their efficacy in improving outcomes and functioning (Weisz, Sandler, Durlak, & Anton, 2005). However, efforts to embed EBTs within community mental health settings often have been unsuccessful. Clinician training-only approaches to bridging the research-to-practice gap have been insufficient; training alone has not been effective in promoting practice change or achieving expected clinical outcomes. Reviews of clinician training highlight the need for post-training technical assistance in the form of consultation or supervision to support implementation of EBTs (e.g., Beidas & Kendall, 2010; Fairburn & Cooper, 2011; Herschell, Kolko, Baumann, & Davis, 2010).

However, supervision, defined as “relationship-based education and training that is work-focused and which manages, supports, develops and evaluates the work of colleagues” (Milne, 2007, p. 439), may be one of the least investigated aspects of EBT implementation in community mental health (Bickman, 2000; Ellis, Krengel, Ladany, & Schult, 1996; Kilminster & Jolly, 2000; Schoenwald, Mehta, Frazier, & Shernoff, 2013). Given that weekly, workplace-based clinical supervision is purportedly a “nearly ubiquitous” infrastructure support in children’s mental health settings (Schoenwald et al., 2008), many have argued that it may offer a cost-neutral EBT support strategy (Schoenwald et al., 2013). Yet, empirical investigations of supervision in community mental health are limited (Schoenwald, Sheidow, & Chapman, 2009). Two distinct but related literatures provide some insight into how supervision in community mental health might support EBT implementation. First, supervision has a long theoretical history as part of professional development and training in clinical psychology (Milne, 2009; Milne & James, 2000). This literature highlights the importance of supervision for facilitating supervisees’ experiential learning, particularly in the areas of case conceptualization, proficiency in clinical techniques, and the clinician-client relationship (Milne & James, 2000; Lambert & Ogles, 1997). Second, the growing theoretical (Nadeem, Gleacher, & Beidas, 2013) and empirical literature on expert consultation as a strategy for supporting EBT implementation is also relevant (e.g., Bearman et al., 2013; Edmunds, Beidas, & Kendall, 2013). This literature demonstrates that receipt of EBT-focused consultation following in-person training predicts greater treatment adoption (Kelly et al., 2000) and competency among clinicians (Miller, Yahne, Moyers, Martinez, & Pirritano, 2004), with higher doses of consultation predicting higher competency (e.g., Beidas, Edmunds, Marcus, & Kendall, 2012). Perceived consultant expertise in the EBT (Schoenwald, Sheidow, & Letourneau, 2004) also appears to be important for clinician adherence.

Workplace-based supervision is an important part of training, and if supervision is to be used to support EBT implementation efforts in community mental health, it likely needs to include a focus on EBT. Milne’s theoretical and empirical work on supervision has mostly focused on graduate training programs, yet, supervision provided in community mental health settings differs in important ways from that provided in graduate programs, given higher caseloads and greater comorbidity in the client population (Southam-Gerow & Kendall, 2016). Similarly, supervision in community mental health differs from expert consultation, which is provided by external, highly knowledgeable “experts,” whose predominant role and focus is EBT- specific support (Nadeem et al., 2013). These differences present challenges for how supervisors in community mental health can best support EBT implementation efforts. In community mental health, supervisors may be overtaxed in the scope of their responsibilities (e.g., Dorsey et al., 2013; Schoenwald, Hoagwood, Atkins, Evans, & Ringeisen, 2010; Schoenwald et al., 2013). EBT coverage likely has to be integrated into supervision that serves a wide range of functions, including oversight of clinical practice or quality of care, administrative responsibilities (e.g., productivity, case documentation, billing procedures), professional development, and personal support for clinicians (e.g., stress management) (Hoge, Migdole, Farkas, Ponce, & Hunnicutt, 2011).

Very little research has examined workplace-based supervision in community mental health within the context of EBT implementation efforts. The available research mostly comes from effectiveness trials of Multisystemic Therapy (MST) (Henggeler, Mihalic, Rone, Thomas, & Timmons-Mitchell, 1998), in which specific aspects of supervision were associated with clinician behavior and client outcomes. In a cross-sectional study, supervisors’ expertise in the intervention (i.e., MST) and in EBT generally was associated with clinician competence (Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002). In another study, supervisors’ focus on adherence to MST treatment principles (e.g., discussion and practice of what the therapist will do in session) predicted clinician adherence, while two distinct aspects predicted change in youth behavior—supervisors’ adherence to the structure and process of the MST supervision protocol and focus on clinician skill development (Schoenwald et al., 2009).

As noted by these authors, MST differs from many other EBTs delivered in community mental health, both in terms of the intervention itself (i.e., a systems-ecological intervention) and in the structure of implementation within organizations (e.g., specific teams with small caseloads, external MST expert consultants who provide ongoing support to workplace-based supervisors, group supervision). In order to better understand how workplace-based supervision in community mental health might support EBT implementation, it is important to explore how supervision is used when organizations are implementing EBTs. What supervision modalities are used (e.g., individual, group, informal) and with what frequency? What functions does supervision serve among supervisors and clinicians trained in EBTs and how is time allocated to the various functions? What functions do supervisors and clinicians perceive as deserving of more time?

Accurso, Taylor, and Garland (2011) examined these questions in a study with 7 supervisors and 12 supervisees in community mental health settings over 130 supervision sessions, although not in the context of EBT implementation and predominantly with supervisees who were trainees (vs. staff clinicians). They found that about 80% of supervision served a clinical function, with the remaining time serving administrative (11%), supervisee professional/academic development (6%), or supervisory relationship (4%) functions. Within the time allocated to clinical functions, about two-thirds focused on case conceptualization and specific therapeutic interventions (Accurso et al., 2011). In our study, we theoretically distinguish case conceptualization and therapeutic interventions from more generic clinical functions (e.g., crisis management, therapeutic alliance) because they may be particularly important for developing EBT competencies in clinicians. This distinction comes both from the literature on supervision for training and professional development that highlights the importance of these two functions (e.g., Milne & James, 2000) and from an empirically derived model of competencies necessary for providing cognitive behavioral treatment for children and adolescents with mood and anxiety disorders (see Sburlati, Schniering, Lyneham, & Rapee, 2011). In their model, Sburlati and colleagues (2011) include case conceptualization as one of three necessary cross-cutting “CBT competencies” and then lists specific CBT techniques, or interventions (e.g., cognitive restructuring, in vivo exposure). A focus on case conceptualization may support clinicians in approaching clients’ presenting problems through an EBT lens, even in the face of client factors that might present challenges to EBT formulation (e.g., foster care; Baker-Ericzén, Jenkins, Park, & Garland, 2015). A focus on intervention techniques provides an opportunity for supervisors to review and plan specifically for what the clinician delivers in session. Given the variability of EBT-focus in graduate training received by clinicians in community mental health (Beidas & Kendall, 2010), clinicians may need more supervision time allocated to case conceptualization and interventions to support experiential learning when attempting to implement EBT. Yet, to date only one small study describes how workplace-based clinical supervision time is spent, and participants were not specifically involved in implementing EBT (Accurso et al., 2011). Examining supervision in the context of EBT implementation is an important next step in identifying ways to leverage supervision to support clinicians in delivering EBTs.

The current study focuses on several aspects of supervision as reported by supervisors and clinicians who are involved in a state-funded EBT implementation effort—the “real world” population expected to integrate EBT support into their existing workplace-based clinical supervision. The first goal was to describe supervision in this sample (i.e., frequency, amount of individual, group, and informal supervision received) to provide more detailed information about the supervision landscape in community mental health. The second goal was to build on the Accurso et al. (2011) study by identifying the perceived general functions of weekly individual supervision and typical time allocated to each function, while also understanding functions to which supervisors and clinicians would like to allocate more time. Potentially, if supervisors and clinicians perceive a greater need to focus on functions that would allow greater support for EBTs (vs. more general clinical functions or non-clinical functions), interventions that target this goal in supervision may be well received. The third goal was to examine factors associated with time allocated to clinical supervision functions and specifically time allocated to the functions that are potentially most relevant to EBT—case conceptualization and intervention.

Method

Background

Data come from a National Institute of Mental Health-funded study of workplace-based clinical supervision provided by supervisors within community mental health organizations who participated in an EBT initiative funded by Washington State (see REMOVED FOR MASKED REVIEW, for the study protocol).

State-funded EBT initiative

In 2007, Washington State began funding an EBT training initiative in Trauma-focused Cognitive Behavioral Therapy (TF-CBT; Cohen, Mannarino & Deblinger, 2006). Organizations were required to have at least one supervisor who participated in the initiative, which includes in-person training and 6 months of post-training expert consultation (1-hour group conference calls, twice monthly). Each year, one to two trainings in TF-CBT are provided—and since 2009, training has also included CBT for depression, anxiety, and behavior problems (3 days of training)— with 100–250 trainees (see REMOVED FOR MASKED REVIEW, for more information). Supervisor-specific post-training supports are available via monthly technical assistance calls and a yearly one-day supervisor training. Organizations are eligible to send teams each year, and as of 2015, 80% of the community mental health organizations in Washington State have participated at least once (i.e., 80 of 99 organizations).

The supervision study collected data from supervisors and clinicians and includes two phases: I) a descriptive study of supervision provided by supervisors trained in TF-CBT as part of the initiative (with no study intervention) and II) a randomized controlled trial (RCT) of two “gold standard” supervision strategy packages selected from efficacy and effectiveness trials. Supervisors and clinicians were enrolled at the beginning of Phase I (fall 2012), the beginning of Phase II (fall 2013), and on a rolling basis as newly eligible participants completed training in TF-CBT. At study entry, all supervisors and clinicians completed baseline self-report surveys on general supervision practices. Only data from these baseline assessments, prior to any supervisor training and RCT activities, are used for this study.

Procedure

All procedures were approved by the Washington State Institutional Review Board. The study team identified organizations that participated in the EBT initiative, were known to be implementing TF-CBT, and had at least one TF-CBT-trained supervisor (N = 33; 75% of the organizations who had participated by 2012, when we began enrollment). We provided supervisors and senior leaders of these organizations with a study overview. Potential supervisors at interested organizations received verbal and written descriptions of the study, and informed consent was obtained prior to the baseline survey. Supervisors who chose to participate identified eligible clinicians from among their supervisees, who were then contacted by our study team and invited to participate. Approximately 76% of the organizations, 73% of the supervisors, and 76% of the clinicians approached consented to participate. All data from these baseline surveys were collected via online Qualtrics surveys between September 19, 2012 and March 10, 2015. Clinicians and supervisors received $30 (enrolled during Phase 1) or $30 and $40 (enrolled during Phase II), respectively, for participation in the baseline survey.

Supervisor participants

Participants were 56 supervisors from 25 community mental health organizations located in 37 separate sites (offices) in Washington State, including rural and urban areas. Criteria for study inclusion were receiving TF-CBT-specific training as part of the EBT initiative (i.e., supervisors were not eligible if they only completed online training) and being a current supervisor of clinicians who were eligible to participate. There were no exclusionary criteria. Table 1 describes characteristics of participating supervisors.

Table 1.

Sample Demographics and Descriptives

	Supervisor (n = 56)		Clinician (n = 207)		Total (N = 263)
Variable	n	%	n	%	N	%
Race/Ethnicity
Asian	2	3.6%	4	1.9%	6	2.3%
Black/African American	0	0.0%	5	2.4%	5	1.9%
Native Hawaiian/Other	1	1.8%	2	1.0%	3	1.1%
White/Caucasian	49	87.5%	170	82.1%	219	83.3%
Multiracial	2	3.6%	9	4.3%	11	4.2%
Hispanic	2	3.6%	15	7.2%	17	6.5%
Other	0	0.0%	2	1.0%	2	0.8%
Female	42	75.0%	174	84.1%	216	82.1%
Academic Degree
Bachelor’s-level	--	--	8	3.9%	8	3.1%
Master’s-level	53	94.6%	193	92.3%	246	93.8%
PhD	1	1.8%	4	1.9%	5	1.9%
PsyD	1	1.8%	1	0.5%	2	0.8%
Other	1	1.8%	1	0.5%	2	0.8%
Training/Degree
Social Work	19	33.9%	49	23.8%	68	26.0%
Psychology	5	8.9%	15	7.2%	20	7.6%
Marriage/Family	9	16.1%	33	16.0%	42	16.0%
Counseling Psyc.	22	39.3%	99	47.8%	121	46.0%
School Psyc.	1	1.8%	1	0.5%	2	0.8%
Other	--	--	10	4.9%	10	3.8%
Licensed***	53	94.6%	90	43.5%	143	54.4%
Orientation
Art Therapy	1	1.8%	3	1.5%	4	1.5%
CBT	38	67.9%	135	65.5%	173	66.0%
Family Systems	10	17.9%	21	10.2%	31	11.8%
Humanistic	2	3.6%	11	5.3%	13	5.0%
Play Therapy	1	1.8%	7	3.4%	8	3.1%
Psychodynamic	1	1.8%	16	7.8%	17	6.5%
Solution-Focused	2	3.6%	8	3.9%	10	3.8%
Other	1	1.8%	6	2.9%	7	2.3%
Uses EBT	42	75.0%	163	78.7%	205	77.9%
Primary Role***
Administrator	7	12.5%	1	0.5%	8	3.0%
Supervisor	43	76.8%	5	2.4%	48	18.3%
Clinician	6	10.7%	201	97.1%	207	78.7%
	M	SD	M	SD	M	SD
Age**	41.7	9.7	37.1	10.6	38.1	10.5
Years in field***	14.3	7.2	7.1	6.1	8.6	7.0
Years at organization***	7.9	6.2	3.5	3.4	4.4	4.5
Years as supervisor	5.5	4.9	--	--	5.5	4.9
Caseload size ***	12.7	11.8	30.9	13.8	27.5	15.2
% caseload children /adolescents	75.0	30.9	79.3	25.6	78.5	26.7
# children w/trauma seen in past 3 months***	4.9	5.7	11.5	9.8	10.1	9.5
# children receiving TF-CBT in past 3 months**	2.66	4.2	5.4	7.6	4.8	7.1

^*p < 0.05

^**p < 0.01

^***p < 0.001

Clinician participants

The clinician sample included 207 clinicians who received supervision from one of the participating supervisors and were trained in TF-CBT (see Table 1). Being trained in TF-CBT was defined either as: 1) receiving training through the initiative (n = 172; 83.1%) or 2) through completion of the freely available, 10-hour, online TF-CBT training program (https://tfcbt.musc.edu) and provision of TF-CBT to one client under supervision by a TF-CBT-trained supervisor at their organization. In-person, initiative training was not required for clinicians because annual training spaces are typically limited to two to three clinicians per organization and therefore, given greater numbers of staff and high turnover rates, organizations regularly rely on the online training. Participants had large caseloads (M = 30.9, SD = 13.8) predominantly comprised of children and adolescents (M = 79.3%, SD = 25.6%). They reported providing TF-CBT to an average of 5.4 (SD = 7.6) children and adolescents in the past three months. Exclusionary criteria were having an adult-only caseload or immediate plans to leave the organization.

Measures

Participant characteristics

Participants provided information on their age, sex, ethnicity, race, education, licensure status, theoretical orientation, TF-CBT training, and other relevant background information (e.g., years at the organization, primary role, EBT use, caseload size). Supervisory-specific background information was also obtained (e.g., number of clinician supervisees, percentage of time doing supervision vs. direct clinical work). TF-CBT training was measured using a summative index of training activities for each participant, from a list of 12 options (e.g., in-person TF-CBT training, read published TF-CBT manual, completed online training, participated in expert consultation, etc.).

Supervision format

Both supervisors and clinicians provided information about supervision format across all cases being supervised (i.e., not only TF-CBT, given our focus on the overall supervision landscape), including modality (group, individual, unscheduled/informal), frequency, and average weekly dose (in minutes).

Functions of supervision

Supervisors and clinicians completed an adapted version of the Supervision Process Questionnaire (SPQ; Accurso et al., 2011) that focused only on individual supervision for all cases supervised. The adapted SPQ asked for the percentage of supervision time spent on nine different supervisory functions for a “typical individual supervision session,” measured as continuous variables (0–100%) that summed to 100% across all functions. The adapted SPQ included five items that are clinically-focused (i.e., therapy intervention/approaches, case conceptualization/formulation, client relationship/alliance building, crisis assessment/management, case management issues), four that are non-clinical (i.e., administrative tasks, supervisee’s professional role, supervisory relationship/process, personal support provided to the clinician), and a write-in “other” option. Clinicians responded individually for their participating clinical supervisor, while supervisors provided a single response for an “average” supervision session across their clinicians participating in the study. Respondents were also asked to rank order the first, second, and third function on which they wish more time was spent. To help characterize these preferences, separate analyses examined respondents’ “first choice” and all “top three” nominated functions (described subsequently).

We computed inter-rater reliability coefficients using two-way random, single-measure, absolute agreement ICCs between clinician and supervisor. The strength of interrater reliabilities varied by construct, which were .55 (ICC_2,1, fair) for clinically-focused items and .71 (ICC_2,1, adequate) for non-clinical items. This level of agreement is fair to adequate when compared to standard benchmarks (e.g., Cichetti, 1994); however, because benchmarks should be used with caution and considered in the context of the data. We view these ICCs to indicate acceptable agreement.because the ICC_2,1 statistic evaluates absolute agreement between raters (i.e. did the raters provide the exact same response values), which is a more conservative estimate than ICC statistics that evaluate the consistency in ranked agreement between raters (i.e. was the overall ranking of values consistent between raters). In our study, the highly specific response options (i.e., percent of time ranging from 0 – 100 for each function) would result in lower absolute agreement ICC statistics as compared to measures with fewer response options (i.e., Likert scale), although psychometric quality might be higher (see Accurso et al., 2011 for more explanation).

EBT attitudes

The Modified Practice Attitudes Scale (MPAS) is a self-report questionnaire used to assess attitudes toward EBT (Borntrager, Chorpita, Higa-McMillan, & Weisz, 2009). We used a five-item version of the MPAS with acceptable internal consistency and good validity (Park, Ebesutani, Chung, & Stanick, 2016). Respondents use a 4-point scale ranging from 0 (not at all) to 4 (to a very great extent) to indicate agreement with statements such as, “Clinical experience and judgment are more important than using evidence-based treatments.” The current study replicated previously reported acceptable internal consistency (Cronbach’s α = .78).

EBT clinician-level activities

The Evidence-based Clinician Checklist is a five-item measure that assesses participant practice behaviors consistent with EBTs, with good established structural validity (see Dorsey, Berliner, Lyon, Pullmann, & Murray, 2016). Items are rated on a 4-point Likert scale (1, rarely; 2, occasionally; 3, regularly; 4, almost always). By design, it includes only essential activities described in a behaviorally specific way to facilitate clarity. Sample items include, “I use standardized measures or questionnaires to identify and measure specific clinical conditions (depression, PTSD, ADHD, behavior problems),” and, “I routinely use a specific evidence-based intervention approach/model or a set of methods from a specific approach that is matched to the identified clinical condition.” The current study replicated previously established good internal reliability (Cronbach’s α = .82). Higher scores represent greater engagement in EBT activities.

EBT implementation climate

The Evidence-Based Organizational Checklist is a six-item questionnaire used to assess the degree to which organizations expect, support, and reward EBT. Content on the six items overlaps with that of other implementation climate measures (e.g., Ehrhart, Aarons & Farahnak, 2014). Items are rated on a 4-point Likert scale (1, never; 2, occasionally; 3, most of the time; 4, ongoing/routine). Sample items for this measure include, “Executive leadership (e.g., administrators, directors) explicitly and repeatedly express support for and promote use of EBT,” and, “Clinicians are provided with EBT training opportunities and ready access to EBT materials (manuals, handouts, equipment).” Unidimensionality of the construct and good internal reliability were established in a prior study (see Dorsey et al., 2016), and good internal reliability was replicated in the current study (Cronbach’s α = .86). Higher scores represent a more supportive EBT implementation climate. Construct validity of the measure is supported by a significantly high office-level Intraclass Correlation ICC(K) of .41. Though ICCs are often a measure of reliability, we state “validity” rather than “reliability” because the clustering of implementation climate ratings by members of the same office supports the idea that the construct is truly rating implementation climate at the higher, office-level (Jacobs, Weiner, & Bunger, 2014; Marsh et al., 2012). ICC(K) was used as a one-way random effects ANOVA because each office was rated by a different set of raters. Given the high ICC and referent for this measure being the office (Marsh et al., 2012), ratings were aggregated to the office-level. The appropriateness of aggregating climate scores at the office-, rather than organization-, level was explored based on participant anecdotal reports and our experience that offices within organizations had unique climates. This was confirmed by comparing office- and organization-level ICC(K), revealing a slightly higher ICC(K) for offices (.41) than organizations (.39).

TF-CBT efficacy

An 11-item index was used to assess clinicians’ report of self-efficacy in TF-CBT. The index is a slight adaptation of a measure created by one of the TF-CBT developers (Deblinger; Child Abuse Research and Service Institute, Rowan University, 2013) and the Project BEST team when conducting a statewide implementation of TF-CBT in South Carolina (National Crime Victims Research and Treatment Center, MUSC, 2010). Participants rate how competent they feel implementing TF-CBT on a 5-point Likert scale (0, not at all; 1, a little bit; 2, somewhat; 3, very much; 4, exceptionally). Sample items include, “Completing trauma narratives with children,” and, “Analyzing complex clinical situations from a TF-CBT perspective.” Cronbach’s alpha in the current sample was .92 and an Exploratory Factor Analysis using maximum likelihood extraction justified retaining a single factor accounting for 56% of the variance.

TF-CBT knowledge

A 13-item, multiple choice knowledge test was used to assess supervisor and clinician knowledge of TF-CBT. The measure builds on the Denver Post Health Survey (Fitzgerald, 2010), with additional items added by our team to assess content similar to that assessed in the TF-CBT certification program (https://tfcbt.org). Average item difficulty was .70 (range .31 – .93), meaning an average of 70% of respondents answered correctly. Average item discrimination was .28 (range .14 – .37), demonstrating a good relation between the probability of answering each item correctly and the respondents’ total score. TF-CBT knowledge was positively associated with the extensiveness of TF-CBT training reported by clinicians (r = .42, p <. 001) and the related, but distinct construct of TF-CBT efficacy (r = .27, p <. 001). We felt these associations supported the convergent validity of this measure.

Analytic Plan

Means, percentages, and standard deviations were calculated for descriptive variables (modality, frequency, and dose of supervision; number of supervisees under each supervisor). T-tests were used to compare supervisor and clinician report of the percentage of time allocated to supervision functions. Chi-square tests were used to compare the proportion of supervisors and clinicians who endorsed each supervision function as the number one function, and one of the top three functions, to which they wished more time could be allocated.

We examined relations between clinician-, supervisor- and organization-level variables and the amount of time spent on supervision functions as reported by clinicians. Null multilevel models (MLMs) were used to examine supervisor-level clustering in clinician-reported percentage of supervision time spent on: 1) five clinical functions overall, and 2) two of the five clinical functions viewed as most EBT-relevant (i.e., case conceptualization and interventions). Because only 14 of the 37 offices and 10 of the 25 organizations had multiple supervisors participating in the study, clustering estimates in three-level models (i.e., office or organization, supervisor, and clinician) were unreliable or failed to converge. Therefore, two-level linear models with supervisor (level-2) and clinician (level-1) were computed using Restricted Maximum Likelihood Estimation with random intercepts for clinician and supervisor, and an identity covariance structure. The office-level aggregated Evidence-Based Organizational Checklist (EBT implementation climate) was therefore included as a variable at Level 2 (supervisor-level).

Model building for hypothesis testing followed standard protocol (Raudenbush and Bryk, 2002). We built models by iteratively testing the bivariate association between level-1 and level-2 predictors and the dependent variable of supervision time spent on the two functions deemed to be most relevant for EBT. Non-normal predictor variables were transformed to achieve normality, as appropriate. Intercepts and slopes were allowed to randomly vary. Best fitting models and variable significance were determined by results from the overall model-2 Log Likelihood, Aikaike Information Criterion Deviance, Schwarz’s Bayesian Criterion χ² statistics (−2L–D, AIC-D, and BIC-D), and the Wald Z statistics. If slope variance was not significant, it was fixed and the model was re-computed.

Results

Supervision in Children’s Community Mental Health

Table 2 presents descriptive results of supervision provided/received. Most supervisors provided both individual and group supervision and over half of the clinicians reported receiving both modalities. Nearly three-fourths of the supervisors (71.7%) and clinicians (71.8%) reported weekly individual supervision that was approximately 1 hour in duration. Almost half of the supervisors and clinicians reported that group supervision occurred weekly (44.2%; 48.3%, respectively) or every other week (34.9; 26.7%), typically for 1 hour. Most clinicians reported receiving informal, unscheduled supervision one to two times a week (49.3%) or three to four times a week (19.3%). On average, supervisors had 7.89 (SD = 4.5) supervisees.

Table 2.

Supervision Received: Modality, Frequency, and Dose

	Supervisor			Clinician
Variable	n	%		n	%
Type of supervision provided/received
Individual only	13	23.2%		85	41.5%
Group only	3	5.4%		17	8.3%
Both	40	71.4%		103	50.2%
Frequency of individual supervision1
Every week	38	71.7%		135	71.8%
Every other week	9	17.0%		29	15.4%
Once a month	2	3.8%		8	4.3%
Other	4	7.5%		16	8.5%
Frequency of group supervision2
Every week	19	44.2%		58	48.3%
Every other week	15	34.9%		32	26.7%
Once a month	6	14.0%		24	20.0%
Other	3	7.0%		6	5.0%
Frequency of informal, unscheduled supervision
Daily	--	--		14	6.8%
3–4 times per week	--	--		40	19.3%
1–2 times per week	--	--		102	49.3%
Every other week	--	--		19	9.2%
Once a month	--	--		15	7.2%
Never	--	--		9	4.3%
Other	--	--		8	3.9%
	M	SD	Range	M	SD	Range
Number of supervisees	7.89	4.5	0–20
Duration of individual supervision1	59.1	3.7	45–71	54.7	12.0	1–90
Duration of group supervision2	74.5	26.4	45–150	67.2	24.1	0–180
Individual supervision minutes/month1				199.2	66.9	25–360
Group supervision minutes/month2				193.2	119.5	30–720
Total (individual & group) minutes/month				282.8	154.4	25–960

¹Excludes those with no individual supervision

²Excludes those with no group supervision

Supervision Functions by Supervisor and Clinician Report

Comparing time spent on clinical functions overall compared to non-clinical functions, supervisors and clinicians reported spending most of their supervision time on clinical functions (69.5% and 69.8%, respectively; see Table 3). Among the nine individual functions (both clinical and non-clinical), therapy interventions received the most supervision time (18.5% and 20.4%, respectively). The second and third individual functions in terms of time allocation were also clinical: case conceptualization/formulation (17.7% and 15.6%, respectively) and therapeutic alliance (13.9% and 16.5%, respectively). The fourth function was non-clinical: administrative (12.8% and 13.2%, respectively). The only statistically significant difference between respondents was that clinicians reported more time spent on therapeutic alliance (M = 16.5%, SD = 11.2%) than supervisors (M = 13.9%, SD = 6.8%), t(145.5) = −2.2, p < .05.

Table 3.

Percent of Supervision Time Allocated to Supervision Functions

	Percentage of time spent on functions
	Supervisor		Clinician
	M (SD)	Range	M (SD)	Range
Clinical Functions
Therapy interventions^	18.5 (8.7)	2 – 40	20.4 (14.3)	0 – 90
Case conceptualization^	17.7 (8.7)	2 – 50	15.6 (10.4)	0 – 60
Therapeutic alliance	13.9 (6.8)*	0 – 30	16.5 (11.2)	0 – 59
Crisis management	10.4 (5.6)	0 – 28	9.3 (7.4)	0 – 60
Case management	9.0 (6.8)	0 – 35	8.2 (7.6)	0 – 60
Non-clinical Functions
Administrative tasks	12.8 (10.7)	0 – 50	13.2 (13.5)	0 – 68
Clinician personal support	8.5 (4.7)	0 – 20	7.6 (7.6)	0 – 60
Supervisees professional role	4.3 (3.1)	0 – 15	4.3 (5.0)	0 – 50
Supervisory relationship	4.6 (3.4)	0 – 15	4.3 (5.2)	0 – 50
Other	0.3 (2.1)	0 – 15	0.8 (6.2)	0 – 70
Overall
Clinical	69.5 (12.5)	0 – 100	69.8 (17.0)	30 – 95
Most EBT relevant	36.3 (11.8)	0 – 95	35.8 (16.6)	4 – 68
Non-clinical	30.2 (12.5)	0 – 100	29.4 (16.5)	5 – 70

^{^}Clinical functions conceptualized as most EBT-relevant

^*Supervisor-Clinician t-test p < 0.05

Functions of Supervision Nominated as Needing More Time in Supervision

Supervisors and clinicians nominated three functions, in order, that they believed needed more time allocation in supervision (see Table 4). Functions of a clinical nature were overwhelmingly nominated as needing more time, with most respondents nominating interventions and case conceptualization as their first choice (among their top three). For the function nominated as the first choice for needing more supervision time, supervisors prioritized case conceptualization (41.1%), followed by therapy interventions (35.7%) and clinician personal support (7.1%). Clinicians prioritized therapy interventions (49.3%), followed by case conceptualization (23.9%) and then clinician personal support (5.7%). The only statistically significant difference in first choice rankings was case conceptualization, with supervisors ranking case conceptualization higher than clinicians, χ²(1, N = 265) = 6.5, p < .05.

Table 4.

Supervision Functions Ranked as the Top Choice and Top Three for Greater Allocation of Supervision Time

	Functions ranked as the first choice for more time allotment		Functions ranked in the top 3 for more time allotment
	Supervisor n (%)	Clinician n (%)	Supervisor n (%)	Clinician n (%)
Clinical Functions
Therapy intervention/approaches	20 (35.7)	103 (49.3)	48 (85.7)	175 (83.7)
Case conceptualization/formulation	23 (41.1)#	50 (23.9)	43 (76.8)	142 (67.9)
Client relationship/alliance	3 (5.4)	11 (5.3)	29 (51.8)#	70 (33.5)
Crisis assessment/management	0 (0)	6 (2.9)	3 (5.4)#	33 (15.8)
Case management issues	0 (0)	3 (1.4)	3 (5.4)#	34 (16.3)
Non-Clinical Functions
Clinician personal support	4 (7.1)	12 (5.7)	2 (3.6)	8 (3.8)
Supervisory relationship/process	3 (5.4)	4 (1.9)	18 (32.1)	66 (31.6)
Supervisees professional role	1 (1.8)	3 (1.4)	6 (10.7)	30 (14.4)
Administrative tasks	0 (0)	0 (0)	12 (21.4)	24 (11.5)
Other	1 (1.8)	5 (2.4)	1 (1.8)	9 (4.3)
Overall
Clinical	46 (82.1)	173 (82.8)	55 (98.2)	195 (93.3)
Most EBT relevant	43 (76.8)	153 (73.2)	55 (98.2)	187 (89.5)
Non-clinical	8 (14.3)	19 (9.1)	31 (55.4)	105 (50.2)

^#Supervisor-Clinician χ² p < 0.05

Examining functions in the top three, compared to clinicians, supervisors were significantly more likely to include therapeutic alliance (51.8% vs. 33.5%, respectively), χ² (1, N = 265) = 6.32, p < .05. Supervisors were significantly less likely to include crisis assessment/management (5.4% vs. 15.8%), χ² (1, N = 265) = 4.10, p < .05, and case management (5.4% vs. 16.3%), χ² (1, N = 265) = 4.38, p < .05. Of the non-clinical functions included in the top three, clinician personal support was endorsed most frequently, by about one-third of supervisors and clinicians.

Factors Associated with Time Allocation to Most EBT-Relevant Functions

Multi-level models indicated that 21% of the variance in clinician report of time spent on clinical supervision functions (i.e., all 5 clinical functions) clustered at the supervisor level (ICC = .207; −2L–D χ² = 10.8, p = .001; AIC-D χ² = 8.08, p = .003). For clinician report of time spent on the two individual functions deemed most relevant to EBT (case conceptualization and therapy interventions), 32% of the variance clustered at the supervisor level (ICC = .318; −2L–D χ² = 18.3, p < .001; AIC-D χ² = 16.3, p < .001). Therefore, clinician’s report of their supervision time may be attributed to supervisor and office/organizational factors in addition to clinician factors. Additionally, as compared to the supervision time spent on general clinical supervision functions, supervision time spent on EBT-relevant functions may be even more attributable to supervisor and office/organizational-level factors.

Model selection

Table 5 depicts results for single-predictor MLMs predicting time spent on EBT-relevant functions. No randomly varying supervisor-level slopes were significant in any models or the models failed to converge, so all supervisor terms were fixed. A null model found significant variance at the supervisor level (τ₀₀=93.34, Wald Z = 2.87, p = .004). Only the measure assessing EBT clinician-level activities (β = 4.51, p = .007) was a significant predictor of time spent on EBT-relevant functions during level-1 bivariate analyses (i.e., clinician-level). In level-2 bivariate analyses (i.e., supervisor-level), significant predictors included EBT implementation climate (β = 8.59, p < .001), supervisor clinical caseload (β = 0.36, p = .019), number of supervisees (β = −0.94, p = .025), time spent doing clinical work per week (β = 0.19, p = .028), and time providing supervision in a week (β = −0.19, p = .032).

Table 5.

Single Factor Level-1 and Level-2 Models Examining Supervision Time Allocated to EBT Most Relevant Functions

Variable	β	SE	p	95% CI
Level 1 (Clinician)
Age	−0.15	0.11	0.171	[−0.36, 0.06]
Sex	−1.55	2.90	0.594	[−7.28, 4.18]
Years in mental health field	−3.02	3.75	0.421	[−10.40, 4.37]
Caseload size	−0.07	0.08	0.427	[−0.23, 0.10]
Total time working with current supervisor	3.34	2.16	0.123	[−0.91, 7.59]
Individual supervision min./month	0.02	0.02	0.330	[−0.02, 0.06]
Group supervision min./month	0.01	0.02	0.339	[−0.02, 0.04]
Total supervision min./month	0.00	0.01	0.889	[−0.02, 0.02]
EBT use	−4.25	2.70	0.117	[−9.57, 1.08]
EBT attitudes	1.37	1.49	.360	[−1.57, 4.30]
TF-CBT knowledge	0.22	0.50	0.666	[−0.77, 1.21]
TF-CBT efficacy	0.74	1.60	0.646	[−2.42, 3.90]
EBT clinician-level activities	4.51	1.65	0.007	[1.27, 7.76]
Turnover intention	−0.22	0.80	0.782	[−1.80, 1.36]
Clinician modification of EBP	−0.64	0.77	0.405	[−2.16, 0.88]
Level 2 (Supervisor)
Age	−0.05	0.18	0.776	[−0.42, 0.32]
Sex	3.28	4.00	0.418	[−4.82, 11.39]
Years in mental health field	−0.02	0.24	0.926	[−0.51, 0.46]
Years as clinical supervisor	−0.07	0.37	0.852	[−0.80, 0.67]
% time providing supervision/week	−0.19	−0.08	0.032	[−0.37, −0.02]
Number of supervisees	−0.94	0.40	0.025	[−1.76, −0.12]
Frequency: individual supervision provision	−4.16	3.32	0.219	[−10.89, 2.57]
Does clinical work (yes/no)	−10.09	5.24	0.062	[−20.70, 0.53]
% time spent doing clinical work/week	0.19	0.08	0.028	[0.02, 0.36]
# of active cases typically carried	0.36	0.15	0.019	[0.06, 0.65]
Participated in WA state supervisor consultation calls	3.58	3.52	0.316	[−3.56, 10.72]
TF-CBT/CBT+ group	−0.81	2.29	0.726	[−5.33, 3.71]
EBT use	0.87	4.25	0.839	[−7.75, 9.49]
EBT clinician-level activities	0.27	2.87	0.925	[−5.54, 6.08]
EBT attitudes	4.48	3.67	0.230	[−2.94, 11.90]
TF-CBT knowledge	0.13	0.70	0.855	[−1.27, 1.53]
TF-CBT supervision efficacy	−2.93	3.66	0.428	[−10.32, 4.47]
EBT implementation climate	8.59	2.16	<0.001	[4.28, 12.89]

Table 6 depicts a few exemplar models from a more extensive model building process (several additional models were constructed but are excluded from Table 6 for space and clarity). When EBT clinician-level activities and EBT implementation climate were included in a single model, neither was significant because of shared variance (r = .51, p < .001). Therefore, the measure of EBT clinician-level activities was removed, leaving no significant level-1 predictors in any of the remaining models. As the sole predictor (Model 1), EBT implementation climate accounted for 36.6% of the supervisor-level variance in the amount of supervision time spent on the two functions most relevant to EBT.

Table 6.

Multilevel Models: Factors Associated with Time Allocated to Most EBT Relevant Functions

Model Comparison	Model 1			Model 2*			Model 3
	β	SE	p	β	SE	p	β	SE	p
Intercept	8.76	6.74	0.197	9.46	9.51	0.326	9.24	9.55	0.340
EBT climate	8.96	2.15	<0.001	10.50	2.78	0.001	9.89	2.91	0.002
# supervisees				−0.70	0.36	0.061	0.11	0.15	0.455
Supervisor clinical caseload							−0.59	0.40	0.147
Variance Components	Variance	SE	p	Variance	SE	p	Variance	SE	p
Residual	182.24	18.23	<0.001	196.46	22.73	<0.001	196.48	22.69	<0.001
Intercept	59.17	25.94	0.23	52.99	25.76	0.040	53.99	26.12	0.039
Model fit					Model 1 to 2 comparison			Model 2 to 3 comparison
					p			p
Model df	4			5			6
−2LL	2133.62			1689.73	<0.001		1691.11	0.239
AIC	2137.62			1693.73	<0.001		1695.11	0.239
BIC	2144.74			1700.34	<0.001		1701.72	0.240

^*Best fitting model

The model building process and goodness-of-fit statistics led to the selection of Model 2 as the final, best-fitting model. Each 1-point increase in EBT implementation climate was associated with a 10.5 percent increase in time spent on the two functions most relevant to EBT. While number of supervisees was not significant (p = .061), a comparison with competing models indicated a significantly better fit, supporting retaining this variable in the final model. Each additional supervisee was associated with a .70 percentage point decrease in the percent of time spent on these two functions. When compared to the null model, level-2 variance was lower, (τ₀₀=52.99, Wald Z = 2.07, p = .040), indicating that the combination of EBT implementation climate and number of supervisees accounted for 43.2% of the supervisor-level variance. None of the models that included any other level-2 predictors that were significant in the bivariate models (percentage of supervision time in a week, supervisor clinical caseload, time spent doing clinical work) had a better fit (see Model 3 as an example).

Discussion

In our sample of EBT-trained supervisors and clinicians, most clinicians received a mix of both individual and group supervision; fewer than half received only individual supervision. Most individual supervision occurred on a weekly basis. Of those receiving group supervision, almost half received it weekly. Individual and group supervision was supplemented with a high frequency of informal, unscheduled supervision. In our examination of functions of individual supervision, consistent with the available literature, workplace-based clinical supervision was used for many functions (Accurso et al., 2011; Hoge et al., 2011; Schoenwald et al., 2013). About 20% of these functions were non-clinical (e.g., administrative, clinician personal support). These functions, while important, reduce the time available for clinical functions overall and for the two specific clinical functions we conceptualized as most relevant to EBT—case conceptualization and interventions. In a supervision hour (the typical duration), our results suggest that about twenty minutes would be spent on these two functions. While individual workplace-based supervision is one hour each week in theory, in many community settings supervision may not actually occur on a weekly basis (see Borders, 2005; Crespi & Dube, 2005), further reducing time for these functions. This was true in our sample, as 20% of the clinicians reported receiving individual supervision only every other week or less frequently. Limited time for overall clinical functions, including those most EBT-relevant, combined with high clinical caseloads in public mental health (e.g., an average of 30 clients per clinician in our sample) likely create challenges for supervisors attempting to support EBT use. Supervisors and clinicians, however, nominated case conceptualization and interventions as functions to which they wished more supervision time could be allocated; suggesting efforts to increase focus in these areas may be positively received.

Our findings have implications for how supervision might be leveraged to better support EBT implementation efforts. First, given variability in supervision modality, frequency, and time allocated to various functions, making an effort to understand how organizations and supervisors currently use supervision is important (e.g., if supervisors establish an EBT-focused supervision group, does that change how they allocate time in individual supervision?). We also need to understand how supervision may support or interact with expert consultation when it is provided as an implementation strategy (Nadeem et al., 2013). Implementation efforts have typically focused on clinicians, but our results, in line with suggestions by Chorpita and Regan (2009), indicate that supervisors need to be included.

Second, our findings provide more support for the importance of implementation climate—the degree to which organizations expect, support, and reward EBT implementation (Klein & Sorra, 1996; Weiner, Belden, Bergmire, Johnston, 2011). Among factors examined that might be associated with time allocated to case conceptualization and interventions, in our final model, implementation climate was the primary predictor, accounting for over one-third of the variability between supervisors. Our results may partially explain other research in which organizational-level factors like implementation climate were more strongly associated with clinicians’ self-reported EBT use than were clinician-level factors (e.g., Beidas et al., 2015).

Interestingly, except for in bivariate models, none of the clinician or supervisor-level characteristics examined (e.g., years as a supervisor, EBT use themselves) were associated with time allocation to EBT-relevant functions despite substantial variance at the supervisor level. Potentially, with increasing demands for supervision to serve multiple functions (e.g., Schoenwald et al., 2013), greater attention to case conceptualization and interventions may only be possible in the context of a positive implementation climate. Although number of supervisees was not significant, its inclusion in the final model improved model fit statistics, suggesting that having more supervisees may play a role in decreasing time allocated to these two functions.

Third, our findings suggest that organizations seeking to increase supervision time focused on clinical functions generally, or specifically on case conceptualization and interventions, may need to explore creative ways to cover the wide range of supervision functions. In our experience in Washington, some organizations have delegated administrative concerns to a non-clinical supervisor or restructured their “all staff” meetings to cover these functions. Others have instituted EBT-focused supervision groups or identified aspects of EBT support that can be led by a peer who has EBT expertise and champions EBT (i.e., peer EBT consultation), but is not a “supervisor” by organizational designation. Recent research suggests that group supervision is a promising efficient avenue for EBT support, given findings that group consultation provided by experts was mostly equivalent to or better than individual consultation (Wiltsey Stirman et al., 2015).

Finally, given our findings that supervisors and clinicians nominated case conceptualization and interventions as top priorities for increased supervision time, research investigating strategies to optimize supervision is needed. One strategy may be for supervisors to closely supervise a small number of cases (or even one case) to whom the clinician is delivering EBT, with the goal of providing the experiential learning deemed critical by Milne and colleagues (see Milne & James, 2000). Potentially, close attention to case conceptualization and specific planned interventions might improve clinician EBT adoption and fidelity not only for the client being supervised, but also for other cases on the clinician’s caseload. This type of case-based learning could extend the reach and impact of even limited supervision time, especially if supervisors use more active strategies like having the clinician behaviorally rehearse EBT intervention components to be delivered, a strategy linked to EBT intervention use in the next session and to analogue fidelity (Bearman, Schneiderman, & Zoloth, 2016; Bearman et al., 2013). Our team is concluding our Phase II RCT testing two supervision conditions that include supervision strategies like behavioral rehearsal that are standard features of supervision in efficacy and effectiveness trials (Beidas & Kendall, 2010), but may be less commonly used in workplace-based clinical supervision (Dorsey et al., 2016). Emerging research is focusing on ways to more practically and efficiently assess and support EBT fidelity, using review of routine clinical materials generated during the delivery of EBTs (e.g., client worksheets) (Wiltsey Stirman et al., 2016). Review of routine clinical materials could support more effective planning for upcoming interventions, and ensuring that planned interventions link to case conceptualization. Each of these areas hold promise to increase the relative proportion of time spent on the most EBT-relevant aspects of supervision. The findings from this study indicate that innovations in this direction would likely be acceptable to clinicians and supervisors.

Limitations

This study has a number of limitations that should be considered. First, the design was cross-sectional and relies on self-report data. Second, participants reported on time allocation in supervision in general (not for a specific supervision session) and at a macro-level, such that participants reported time on “interventions” and not specifically on EBT-focused interventions and case conceptualization. We view our approach as providing a liberal estimate of time available for EBT-focused case conceptualization and interventions, given that they would have to fit within overall time allocated to these functions. Third, while results have the potential to be highly generalizable due to the relatively large, diverse, and representative sample of community mental health organizations that participated, Washington State may have a unique service context that could differentially impact supervision and/or overall uptake of EBT. Even with these limitations, however, our study is one of the few that examines supervision provided by community-based clinical supervisors involved in EBT implementation efforts.

Conclusions and Future Directions

These findings offer important implications. Supervision is provided via a range of modalities, with individual supervision covering a wide range of functions. More research is needed on how to support supervisors and organizations in providing ongoing support for EBT implementation; however, it should be grounded in an understanding of how time is currently allocated, as well as in the various ways that supervisors and organizations deliver supervision. Findings from the current study suggest that both supervisors and clinicians may be interested in finding ways to spend more of their supervision time on clinical functions and specifically on functions most relevant to EBT. However, it is also clear that any increased focus on particular functions likely means that other functions (e.g., clinician personal support, case management, administrative) will receive less time. As a field, we need to partner with clinicians, supervisors, and organizations to identify necessary elements of clinical supervision that support positive outcomes at the clinician and client-level. Given the multiple demands placed on supervisors, it may be necessary to find creative ways to allow supervisors to focus more of their supervision on clinical aspects—particularly when supporting clinicians in gaining competence in EBT—without entirely supplanting other functions that are perceived as important (i.e., personal support for clinicians) or that may be necessary for operations (administrative functions).