The Role of Setting Versus Treatment Type in Alliance within Youth Therapy

Abstract

Objective

Does the strength of the youth-therapist alliance differ across treatment settings or treatment type? We examined these questions in the context of youth therapy

Method

89 youths (M age = 10.56, SD = 1.99; 63.70% Caucasian; 52.80% male) diagnosed with an anxiety disorder received (a) manual-based individual cognitive-behavioral therapy (ICBT) in a research setting, (b) manual-based ICBT in practice settings, or (c) non-manualized usual care (UC) in practice settings. Coders, using the Therapy Process Observational Coding System–Alliance scale, rated 865 sessions. Youth completed the Therapeutic Alliance Scale for Children at posttreatment.

Results

Youth who received ICBT in a research setting had significantly higher observer-rated alliance than youth who received either therapy delivered in practice settings. In practice settings, youth who received ICBT had significantly stronger observer-rated alliance early in treatment than youth in UC, but this difference was not observed at the end of treatment. Similarly, youth-report alliance at post-treatment was significantly higher in ICBT in the research setting, and there was no difference between ICBT and UC delivered in practice settings. Alliance differences largely held when controlling for youth characteristics; however, differences early in treatment between the ICBT groups were no longer statistically significant when controlling for anxiety severity or primary anxiety diagnosis.

Conclusions

Our findings suggest that (a) the alliance may be stronger in research settings, and (b) treatment manuals do not undermine alliance. Future research is required to help pinpoint whether other youth, therapist, or setting factors contribute to the lower alliance seen in practice settings.

Efforts to disseminate and implement evidence-based treatments (EBTs) for youth face a number of challenges. Some have raised concerns that certain characteristics of EBTs (Addis & Krasnow, 2000; Nelson, Steele, & Mize, 2006) and/or community-based service settings (e.g., organizational climate, productivity demands; Glisson et al., 2008; Green, Albanese, Cafri, & Aarons, 2014) may serve to weaken the youth-therapist alliance. If certain characteristics of EBTs and/or settings interfere with the alliance, this could reduce the appeal of EBTs for both clinicians and their clients and thus represent a potential barrier to expanding use of EBTs in community-based service settings.

Determining if certain characteristics interfere with the alliance represents an important goal for the field because the alliance may be particularly important in youth therapy, as youth rarely self-refer (Karver, Handelsman, Fields, & Bickman, 2005). Indeed, a therapist’s ability to develop an alliance with a youth—one that is marked by positive affect and participation in therapeutic activities—is considered to be an important ingredient of therapy (Chu et al., 2004; Elvins & Green, 2008). A strong youth-therapist alliance (hereafter called the alliance) is associated with positive outcomes, such as reduced symptomatology and treatment attendance, across different problem types and treatments (McLeod, 2011; Shirk, Karver, & Brown, 2011). Thus, understanding factors that may influence alliance in youth therapy represents an important research objective.

Some have argued that manual-guided EBTs may interfere with alliance formation (Addis & Krasnow, 2000; Nelson et al., 2006) due to a lack of flexibility to address individual client needs (e.g., Henry, Strupp, Butler, Schact, & Binder, 1993; Pagoto et al., 2007). To our knowledge, only two previous studies have addressed this issue related to youth therapy. Langer, McLeod, and Weisz (2011) compared the quality of the alliance in manual-guided EBTs and non-manualized treatment delivered to 76 youth in community-based service settings (herein called practice settings).¹ Early in treatment the alliance was significantly higher in manual-guided treatment, but the two groups converged over time. Ormhaug, Jensen, Wentzel-Larsen, and Shirk (2014) compared the quality of the youth-rated alliance in a manual-guided EBT and a non-manualized treatment delivered to 156 traumatized youth in practice settings. There were no significant differences in the youth-rated alliance between the treatments at session 1 or 6. These findings suggest that the quality of the alliance may not differ significantly across manual-guided and non-manualized treatment in practice settings. However, both studies lacked some important methodological elements that could help address key alternative explanations; for example, the studies did not include a control for treatment setting.

The alliance may also differ when an EBT is delivered in practice settings, rather than university-based research clinics (herein called research settings). A case could be made that alliance might be stronger in research settings. Therapists in research settings are often given time to prepare for sessions, which could focus on alliance building activities. Moreover, youth may be more motivated to participate and form an alliance given that families often seek out the brand of therapy provided in specialty research settings. By contrast, therapists in practice settings work under productivity requirements that may leave little time to prepare for sessions (Gray, Elhai, & Schmidt, 2007; Nelson et al., 2006) and may face other challenges (e.g., the need to take cases on rotation rather than focus on youth who fit the therapists’ skills best) that could interfere with alliance building (Weisz & Addis, 2006). The greater youth demographic and clinical diversity seen in practice settings (Ehrenreich et al., 2011; Southam-Gerow, Chorpita, Miller, & Gleacher, 2008) may also negatively impact the alliance. However, it is also plausible that the alliance may be higher in practice settings. In practice settings, the importance that therapists place on the alliance (Addis & Krasnow, 2000; Nelson et al., 2006) may lead them to emphasize alliance building activities over other elements of treatment delivery. Conversely, therapists in research settings may focus on delivering the components of the EBT (i.e., achieving high treatment adherence) to the detriment of the alliance.

Although it is plausible that setting may influence alliance, empirical tests are lacking. If there were a difference in the alliance across settings, this could serve as a focus for quality improvement efforts. We thus investigated whether the alliance differs when the same manualized EBT is delivered across research and practice settings.

To adequately address questions about alliance requires careful consideration of measurement options. Two broad strategies for alliance assessment warrant attention: self- and observer-report. Each method has certain advantages and both methods are used in the present study. Self-reports provide valuable data on the treated youth’s subjective perceptions of the alliance (Elvins & Green, 2008; Shirk & Saiz, 1992). Because a significant proportion of the research examining alliance has used self-report measures (see McLeod, 2011), including such instruments supports comparisons with existing literature. Accordingly, in the present study, we examined alliance at post-treatment using the Therapeutic Alliance Scale for Children (TASC; Shirk & Saiz, 1992), the most widely-used and thoroughly studied youth-report instrument of alliance (see Accurso, Hawley, & Garland, 2013; McLeod, 2011).

Observer-rated methods also provide advantages. Observer-rated alliance instruments are not subject to youth’s varying ability to report on the alliance relationship (Shirk & Karver, 2003). Observer-rated data also avoid the confounding effects of demand characteristics (pressure to say nice things about a therapist) that can complicate interpretation of findings (Shirk & Karver, 2003; Shirk & Russell, 1998) and contribute to ceiling effects (Chu et al., 2004). In the present study we used the Therapy Process Observational Coding System for Child Psychotherapy – Alliance Scale (TPOCS-A; McLeod & Weisz, 2005), a widely researched, observer-rated instrument of alliance (see Fjermestad et al., 2012; McLeod, 2011).

Another important issue in alliance assessment is timing of measurement within the treatment episode. It is useful to assess alliance early in treatment to avoid potential confounding with symptom improvement (Kazdin, 2007). It is also useful to assess alliance at multiple time points, as research suggests that the alliance may shift over the course of therapy (e.g., Chu, Skriner, & Zandberg, 2014; Hudson et al., 2014; Kendall et al., 2009). Kendall et al. (2009), for example, found that in family-focused CBT for youth anxiety, the therapist-, youth-, mother-, and father-rated alliance gradually increased and then leveled off. More recently, Chu and colleagues (2014) found that in individual CBT (ICBT) for youth anxiety, therapist-report alliance increased and then leveled off, but youth-report alliance was stable over the course of treatment. Interestingly, a different pattern has been found for observer-rated instruments. Hudson and colleagues (2014) found that in ICBT for youth anxiety, observer-rated alliance showed a slight downward slope. Given the emerging evidence that alliance may change over treatment, it seems important to assess whether differences in the alliance exist over time across treatments and settings.

We evaluated the quality of the alliance with youth diagnosed with anxiety disorders who received the same manualized EBT (i.e., Coping Cat for youth diagnosed with anxiety disorders; Kendall & Hedtke, 2006a, 2006b) in research or practice settings, or non-manualized UC. In the research setting, youth were referred to an efficacy trial and treated by research therapists. Youth in the practice settings were clinically-referred and treated by therapists employed by the clinics; therapists volunteered to participate in the effectiveness trial and were randomly assigned to the EBT or UC group. Given limited prior literature in this area and competing theories about the impact of manuals on alliance, we tested two competing hypotheses. First, we tested the hypothesis that the alliance would be significantly higher in manual-guided ICBT than non-manualized UC due to the structure and focus provided by the ICBT program. Second, we tested the hypothesis that the alliance would be significantly higher in non-manualized UC than manual-guided ICBT due to the client-centered nature of UC. Because no prior published report has examined whether the alliance differs when an EBT is delivered in a research and practice setting, we offer no a priori hypothesis about the EBT delivered in the research setting.

Method

Participants

Participants were 89 youths (M age = 10.56, SD = 1.99; range 7–15 years; 63.70% Caucasian; 52.80% male) with a primary anxiety disorder who participated in one of two randomized clinical trials (RCTs) conducted in a research or practice setting. This project utilized archived session recording data. To be included, youth had to have at least two audible sessions and have received therapy from a single therapist. All procedures were IRB approved. See Tables 1 and 2 for descriptive information about the youth and therapist participants.

Table 1.

Youth Descriptive Data and Comparisons Across Groups

Variable	M (SD) or %			F or Chi Square	p
	ICBT (N = 51)	YAS-ICBT (N = 17)	YAS-UC (N = 21)
Age	10.36 (1.90)	11.32 (2.32)	10.44 (1.91)	1.56	.217
Sex
Male	60.80	29.40	52.40	5.04	.081
Race/Ethnicity				29.91	<.001
Caucasian	86.28a	41.20	33.30
African-American	9.80	-	9.50
Latino	1.96	17.60b	42.90c
Mixed/Other	1.96	5.90	9.50
Not Reported	-	35.30d	4.80
CBCL
Total	63.18 (8.44)	64.19 (7.34)	65.06 (6.46)	0.39	.678
Internalizing	67.40 (8.37)	66.38 (8.33)	66.82 (8.33)	0.10	.904
Externalizing	52.96 (10.08)	60.81(7.49)b	59.41 (9.67)c	5.61	.005
Anxious-Depressed	63.10 (9.31)	68 (8.69)b	68.29 (8.37)c	3.51	.034
Primary Diagnoses				22.73	.001
GAD	37.30a	5.90	14.30
SAD	29.40	35.30	38.10
SOP	33.30	23.50	28.60
SP	-	35.30d	19.00
Family Income				15.66	<.001
Up to 60k per year	35.30	70.60b	76.20c
Number of Sessions	15.92 (1.43)	16.82 (5.02)	15.71 (9.34)	0.26	.775
Weeks in Treatment	19.52 (3.97)	26.38 (10.41)b	26.84 (15.53)c	6.45	.002
Number of Coded Sessions	10.43 (2.84)	12.47 (4.61)	10.00 (6.00)	1.94	.145

Note. ICBT = individual cognitive-behavioral therapy delivered in Kendall et al. study, YAS-ICBT = ICBT delivered in YAS, YAS-UC = usual care delivered in YAS. CBCL = Child Behavior Checklist, GAD = generalized anxiety disorder, SAD = separation anxiety disorder, SOP = social phobia, SP = specific phobia. Analysis of variance was conducted with continuous variables whereas chi-square analyses were conducted with categorical variables.

^aICBT > YAS-ICBT, YAS-UC.

^bYAS-ICBT > ICBT.

^cYAS-UC > ICBT.

^dYAS-ICBT > ICBT, YAS-UC.

Table 2.

Therapist Descriptive Data and Comparisons Across Groups

Variable	M (SD) or %			F or Chi Square	p
	ICBT (N = 16)	YAS-ICBT (N = 13)	YAS-UC (N = 16)
Sex
Male	12.50	15.40	12.50	.104	.949
Race/Ethnicity				12.59	.05
Caucasian	81.25	53.80	43.75
African-American	-	-	-
Asian American	6.25	15.40	-
Latino	6.25	15.40	37.50
Mixed/Other	-	15.40	12.50
Not Reported	6.25	-	6.25
Professional Training				12.40	.015
Psychology	100.00a	38.40	37.50
Social Worker	-	30.80b	25.00c
Other	-	30.80b	31.25c
Not Reported	-	-	6.25

Note. ICBT = individual cognitive-behavioral therapy delivered in Kendall et al. study, YAS-ICBT = ICBT delivered in YAS, YAS-UC = usual care delivered in YAS.

^aICBT > YAS-ICBT, YAS-UC.

^bYAS-ICBT > ICBT.

^cYAS-UC > ICBT.

University-based research setting

Kendall et al. (2008) compared the efficacy of ICBT, family-CBT, and a family-based education/support/attention control group. Only ICBT was used in this study. The 51 youth participants (M age = 10.36, SD = 1.90; 86.28% Caucasian; 60.80% male) received therapy at a specialty university-based research clinic for the treatment of youth anxiety disorders. Therapy was delivered by clinical psychology doctoral trainees and licensed clinical psychologists (N = 16; 12.50% male; 100% psychologists) who were 81.25% Caucasian, 6.25% Latino, and 6.25% Asian/Pacific Islander (6.25% did not report). At post-treatment, 64% of participants no longer met criteria for their primary anxiety disorder and the youth had a mean CBCL Internalizing score (mother report) of 62.17 (SD = 8.30).

Community-based service settings

The Youth Anxiety Study (YAS; Southam-Gerow et al., 2010) compared the effectiveness of ICBT (YAS-ICBT; N = 17) to UC (YAS-UC; N = 21). The 38 youth participants (M age = 10.83, SD = 2.12; 36.80% Caucasian; 42.10% male) were clinically-referred and received therapy at community mental health clinics in Los Angeles county. Therapists were clinic employees (N = 29) who volunteered to participate in the study and were randomly assigned to groups. YAS-ICBT therapists (N = 13; 15.40% male) were 53.80% Caucasian, 15.40% Latino, 15.40% Asian/Pacific Islander, and 15.40% mixed/other. Professionally, 30.80% were social workers, 38.40% psychologists (23.10% masters’ level; 15.30% doctoral level), and 30.80% reported “Other.” YAS-UC therapists (N = 16; 12.50% male) were 43.75% Caucasian, 37.50% Latino, and 12.50% “Mixed/Other” (6.25% did not report). Professionally, 25.00% were social workers, 37.50% psychologists (31.20% masters’ level; 6.30% doctoral level), and 31.25% “Other” (6.25% did not report). At post-treatment, 66.70% and 73.70% of the youth no longer met diagnostic criteria in the YAS-ICBT and YAS-UC groups, respectively. And at post-treatment the youth had a mean CBCL Internalizing score of 58.87 (SD = 9.03) in YAS-ICBT and 56.12 (SD = 10.36) in YAS-UC.

Treatments

CBT

Therapists in ICBT and YAS-ICBT delivered Coping Cat, an ICBT program designed for youth diagnosed with anxiety disorders (Kendall & Hedtke, 2006a, 2006b). Coping Cat includes 16 sessions (14 conducted individually with the youth; 2 with parents). The program emphasizes anxiety management skills training (e.g., relaxation, problem-solving) and exposures. Homework is regularly assigned to the youth. In both studies, the gold standard in training for RCTs was used to train the therapists (i.e., training workshop, treatment manual, and supervision; Sholomskas et al., 2005). The therapists reviewed the Coping Cat therapist manual, attended a training workshop lead by experts in the Coping Cat program, and participated in regular supervision with an expert in the Coping Cat program. Adherence to the Coping Cat, measured with the Coping Cat Brief Adherence Scale (see Kendall, 1994; Kendall et al., 1997), was found to be high in ICBT and YAS-ICBT. A treatment differentiation check determined that the YAS-ICBT and YAS-UC groups did not overlap significantly in CBT interventions unique to Coping Cat. (See Kendall et al. (2008) and Southam-Gerow et al. (2010) for more details about treatment integrity).

Usual care (UC)

Therapists who provided UC agreed to use the therapeutic interventions they regularly delivered and believed to be effective in their routine practice. UC therapists received clinical supervision provided as standard practice in their clinic. Therapists in UC adopted a multifaceted approach to treatment and delivered a broad range of therapeutic interventions consistent with multiple theoretical orientations. Overall, the therapists used client-centered and family-focused interventions the most; psychodynamic and CBT interventions were used but to a lesser degree (McLeod, Smith, Southam-Gerow, Weisz, & Kendall, 2015; Southam-Gerow et al., 2010).

Alliance Instruments

Therapy Process Observational Coding System for Child Psychotherapy-Alliance scale (TPOCS-A; McLeod & Weisz, 2005).

The TPOCS-A consists of 6 items that assess affective elements of the client–therapist relationship, and 3 items that assess client participation in therapeutic activities. Coders observe entire sessions and rate each item on a 6-point scale ranging from 0 (not at all) to 5 (a great deal). The TPOCS-A has demonstrated item interrater reliability ranging from .48 to .80 (M ICC = .67), internal consistency ranging from .91 to .95 (M α = .92), convergent validity with self-report alliance instruments ranging from .48 to .53 (Fjermestad et al., 2012; Liber et al., 2010), and predictive validity in relation to outcome (e.g., McLeod & Weisz, 2005; Liber et al., 2010). In this study, interrater reliability, ICC(2,2), for the TPOCS-A scale was .82; internal consistency was .81.

Therapeutic Alliance Scale for Children (TASC; Shirk & Saiz, 1992).

The TASC was used to assess youth-rated alliance. Items are scored on a 4-point scale ranging from 1 (not true at all) to 4 (very true). The TASC has evidenced convergent validity with other alliance instruments (Accurso et al., 2012; Fjermestad et al., 2012) and predictive validity in relation to outcome (Hawley & Weisz, 2005; Kazdin, Marciano, & Whitley, 2005). In YAS, the 7-item version (Shirk & Saiz, 1992) was administered whereas the revised 12-item version was administered in Kendall et al. (2008). Six items had identical wording, so scores were based on a sum of those 6 items. These 6 TASC items focus on the bond dimension of the alliance (e.g., “I liked spending time with my therapist”; “I felt like my therapist was on my side and tried to help me”). For analyses, we used TASC scores collected at post-treatment.² The TASC has demonstrated acceptable psychometric properties in previous studies (e.g., Accurso et al., 2012; Fjermestad et al., 2012; Shirk & Saiz, 1992); internal consistency for the 6 items utilized was α = .81–.86 across samples.

Symptom Instrument

Child Behavior Checklist

(CBCL; Achenbach, 1991). The CBCL is a widely-used and psychometrically sound instrument. The 118 items assess symptoms across a broad range of clinical significance (e.g., shyness, suicide attempts). CBCL raw scores are converted to T-scores for three broad-band scales (e.g., Internalizing), and eight narrow-band sub-scales (e.g., Somatic Complaints). In the current investigation, three CBCL scales were used: Internalizing, Externalizing, and Anxious-Depressed. The CBCL has been used with acceptable levels of reliability to measure behavior problems of youth aged 4–16 in a variety of cultural settings (Achenbach, 1991; Achenbach et al., 1990).

TPOCS-A Coding and Session Sampling Procedures

Two doctoral student coders, naïve to study group, were trained over a 3-month period, followed by weekly meetings to prevent coder drift. Sessions were randomly assigned to coders. Each session was double coded and the mean score was used in analyses to reduce measurement error. All sessions for each case were coded except for the first and last session, as these sessions may contain intake or termination content. Recordings were not rated if (a) audible content was shorter than 15 minutes, (b) less than 75% of the dialogue was in English, or (c) missing or damaged. Of the 1428 sessions held, 954 (67%) were rated (66% ICBT (n = 532), 75% YAS-ICBT (n = 212), 67% YAS-UC (n = 210)). There were no significant differences across groups in the percent of sessions coded, nor were there significant differences across groups in the percent of sessions coded from the first and second half of treatment. This study focused on the subset of sessions that the youth attended (N = 865; ICBT n = 463; YAS-ICBT n = 197; YAS-UC n = 205).

Data Analysis Plan

Analyses of group differences in alliance were conducted using multilevel modeling (Raudenbush & Bryk, 2002) using HLM 7.01 (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011) to account for the nesting of youth in therapists and repeated measures in youth (TPOCS-A only). Prior to data analysis, missing data patterns were examined. As discussed above, numbers of sessions coded did not vary across conditions and multilevel models can accommodate varying amounts of repeated measures data across participants, so these analyses focused on patterns of missingness in the youth-level control variables and TASC data. Rates of missing data were 10.2% (TASC data) or less across variables. These data were missing completely at random (Little’s MCAR test Χ² = 181.23, DF = 152, p = .053) and missingness on individual variables was not significantly related to TASC or TPOCS-A scores. Analyses involving the youth-level variables (i.e. TASC analyses and the analyses examining youth-level characteristics described below) were conducted using the multiple imputation function in HLM 7.01, using 10 datasets imputed using IBM SPSS Statistics version 22.0.

Unconditional three-level models were first fit to the TPOCS-A data; 28% of the variance in alliance ratings were within participant (level 1), 46% was at the youth level (level 2), and 25% at the therapist level (level 3), indicating use of three-level multilevel analyses was appropriate. Given differences in the shape of change in alliance documented in previous studies (e.g., Chu et al., 2014; Hudson et al., 2014; Kendall et al., 2009), additional models were fit to the data examining whether change over time was linear, quadratic, or loglinear. The linear model, predicting each session’s TPOCS-A score from time (measured as weeks in treatment given variable session spacing across groups), found a significant slope (γ₁₀₀ = −.0054, p < .05) and a comparison of model deviance statistics indicated the linear model improved fit over the unconditional model (Χ² = 20.47, df = 3). Adding a quadratic term did not improve model fit (Χ² = 4.09, df = 4) and fitting a loglinear model found that slope was not significant, indicating that a linear model was most appropriate. The residuals file from the linear model was then used to identify outliers on intercept and slope. One case was an intercept outlier; this case was excluded because first session differences between this case and others were not likely due to group. Two other cases were outliers on slope; however, given that both cases were in ICBT, these cases were retained because they might represent true effects.

For the TPOCS-A, group differences were examined by dummy coding group membership and entering it as a youth-level (level 2) predictor of intercept (level of alliance at the first treatment session) and slope (linear change in alliance over time in weeks; TIMEWEEK in the model below); this model was run twice, once with YAS-UC as the reference group and once with YAS-ICBT as the reference group, allowing for all pairwise comparisons. Given the variability in treatment dose across groups, the number of sessions each youth received was included as a level 2 control variable (DOSESESS in the model below). For example, the models with YAS-UC as the reference group were constructed as:

• L1: TPOCS-A_ijk = π_0jk + π_1jk*(TIMEWEEK_ijk) + e_ijk

• L2: π_0jk = β_00k + β_01k*(ICBT_jk) + β_02k*(YAS-ICBT_jk) + β_03k*(DOSESESS_jk-DOSESESS..) + r_0jk

        π_1jk = β_10k + β_11k*(ICBT_jk) + β_12k*(YAS-ICBT_jk) + β_13k*(DOSESESS_jk-DOSESESS..) + r_1jk

• L3: β_00k = γ₀₀₀ + u_00k

        β_01k = γ₀₁₀

        β_02k = γ₀₂₀

        β_03k = γ₀₃₀

        β_10k = γ₁₀₀

        β_11k = γ₁₁₀

        β_12k = γ₁₂₀

        β_13k = γ₁₃₀

Where γ₀₁₀ is the estimated difference on the TPOCS-A between the ICBT and YAS-UC groups at the first session, γ₀₂₀ is the estimated difference between the YAS-ICBT and YAS-UC groups at the first session, γ₁₁₀ is the estimated difference in slope between the ICBT and YAS-UC groups, and γ₁₂₀ is the estimated difference in slope between the YAS-ICBT and YAS-UC groups. Analysis of the TASC was similar, except 2 level models (youth nested within therapists) were used to predict post-treatment TASC values and multiple imputation was applied (see above).

Effect sizes were computed in two ways. For group differences in intercepts (i.e., group mean differences), the group difference parameter estimate (e.g., γ₀₁₀, above) was divided by the raw data standard deviations. For group differences in slopes, effect sizes were computed following Feingold (2009), who recommended multiplying the parameter by the length of treatment and dividing it by the raw data standard deviation. As the length of treatment varied across groups (see Table 1), the average length of treatment of 23 weeks was used in this computation. Both forms of effect size computations are considered estimates of Cohen’s (1988) d, where .2 is considered small, .5 medium, and .8 large.

Finally, we examined whether the group differences held when controlling for youth characteristics that differed between the groups. Control variables were tested simultaneously and entered using grand mean centering. These analyses were also conducted using multiple imputation (see above).

Results

Our first step was to conduct sample bias analyses to determine if the 89 youths and therapist participants selected for this study differed from the participants in the parent studies (Kendall et al., 2008; Southam-Gerow et al., 2010). We only found one difference for youth demographic characteristics. There was a lower proportion of African-American youth (0% vs 16.67%) and a higher proportion of Caucasian youth (41.18% vs. 29.17%) in our YAS-ICBT sample compared to the parent study, Χ²(3, N = 24) = 11.53, p = .009).

As detailed in Table 1, we examined differences between groups (ICBT, YAS-ICBT, YAS-UC) on several youth demographic, baseline clinical, and treatment characteristics. Groups differed on race/ethnicity, level of externalizing symptomatology, level of anxiety and depressive symptomatology (as measured by the CBCL Anxious-Depressed subscale), primary anxiety disorders, family income level, and weeks in treatment. As reported in Table 2, groups did not differ on therapist sex or race/ethnicity, but did differ on therapist professional training. Descriptive statistics for the TPOCS-A and TASC scores for the three groups are presented in Table 3. These data indicate that the TPOCS-A and TASC scores were significantly higher in ICBT. The correlation between the endpoint TPOCS-A and the TASC was .38 (p < .01).

Table 3.

Means and Standard Deviations for Alliance

Alliance measure	ICBT		YAS-ICBT		YAS-UC		F	p
	M	SD	M	SD	M	SD
TPOCS-A	3.41a,b	.37	3.16	.30	2.94	.37	13.74	< .0001
TASC	22.10a,b	3.10	18.50	5.35	18.56	4.35	8.16	.001

Note. The TPOCS-A scores were produced by calculating the mean score for all sessions that the youth attended from each case. TPOCS-A = The Therapy Process Observational Coding System for Child Psychotherapy – Alliance scale; TASC = Therapeutic Alliance Scale for Children. ICBT = individual cognitive-behavioral therapy delivered in Kendall et al. study; YAS-ICBT = ICBT delivered in YAS; YAS-UC = usual care delivered in YAS.

^aICBT > YAS-ICBT

^bICBT > YAS-UC

Analyses of group differences on the TPOCS-A are presented in Table 4. First, a linear model, with time centered at the first session, was fit to the data. Across all participants, alliance was estimated to be 3.28 at the first session (i.e., the intercept; p < .001) and there was a significant decrease in alliance over time (γ₁₀₀= −.0048, p = .033). Fifty-four percent of the variability in intercept was at the therapist level and 46% at the youth level; 99% of the variability in slope was at the youth level, so all slope effects were fixed at level 3 in subsequent models.

Table 4.

Multilevel Models of Alliance Between Groups

	Coefficient	S.E.	ES	Deviance1	N Parameters in Model	AIC
TPOCS-A analyses
Linear Model				803.00	7	820.00
Intercept (first session value), γ000	3.29***	.055	N/A
Slope (change over time in weeks), γ100	−.0054*	.0020	N/A
Models including group contrasts1				771.42	13	797.42
Group Differences in Intercept
ICBT (1) vs YAS-UC (0)	.57***	.09	1.18
YAS-ICBT (1) vs YAS-UC (0)	.34**	.11	.70
ICBT (1) vs YAS-ICBT (0)	.22*	.10	.45
Group Differences in Slope
ICBT (1) vs YAS-UC (0)	−.0077	.0051	−.37
YAS-ICBT (1) vs YAS-UC (0)	−.011*	.0053	−.52
ICBT (1) vs YAS-ICBT (0)	−.0031	.0050	−.15
TASC analyses
Unconditional Model
Intercept (post-treatment value), γ00	20.23***	.70	N/A
Models including group contrasts1
Group Differences in Intercept
ICBT (1) vs YAS-UC (0)	3.39**	1.13	.83
YAS-ICBT (1) vs YAS-UC (0)	−1.62	1.44	−.40
ICBT (1) vs YAS-ICBT (0)	5.01***	1.19	1.22

Note. For parsimony, this table includes only parameters of central conceptual interest (e.g., intercepts are not reported for the models examining group differences, parameters are not reported for the control variable, number of sessions).

¹Analyses of the TASC were completed using multiple imputation. Deviance statistics are not available in the HLM software for the aggregated models from such analyses.

S.E. = Standard Error; ES = effect sizes (Cohen’s d)

^*p < .05

^**p < .01

^***p < .001

ICBT = individual cognitive-behavioral therapy delivered in Kendall et al. study; YAS-ICBT = ICBT delivered in YAS; YAS-UC = usual care delivered in YAS; TPOCS-A = Therapy Process Observational Coding System for Child Psychotherapy – Alliance scale; TASC = Therapeutic Alliance Scale for Children.

Next, group was entered as level 2 predictors of both intercept and slope. At the first session, all groups differed from one another on the TPOCS-A, with alliance being higher for ICBT than for YAS-ICBT (γ = .22, p = .025) and YAS-UC (γ = .57, p < .001), and YAS-ICBT having significantly higher alliance than YAS-UC (γ = .34, p = .003). The only significant slope difference was between the YAS groups, with YAS-ICBT showing steeper declines over treatment than YAS-UC (γ = −.011, p = .026).

Analysis of the TASC data followed the same sequence of steps (see Table 4). The unconditional model indicated that the average predicted score on the TASC across groups was 20.82, with 67% of the variance at the youth level. When group was entered into the model, ICBT participants had significantly higher TASC scores than both YAS-ICBT (γ = 5.01, p < .001) and YAS-UC (γ = 3.39, p = .005), but there was not a statistically significant difference between the two YAS groups. Because the TPOCS-A had been modeled to compare mean levels of alliance at the first session, and the group differences found in those analyses differed from those found in the analysis of the TASC, we conducted a post-hoc analysis, re-centering the TPOCS-A at the last session and examining group differences in this new model intercept. These group differences paralleled the TASC analyses, with the ICBT group having significantly higher TPOCS-A scores at the last session than both the YAS-ICBT (γ = .34, d = .70, p = .005) and YAS-UC groups (γ = .51, d = 1.05 p < .001) and no statistically significant differences between the two YAS groups (γ = .17, d = .35, p = .198).

Finally, we examined whether the group differences held when controlling for youth characteristics that differed between the groups (i.e., youth race/ethnicity, CBCL Externalizing score, CBCL Anxious-Depressed score, primary diagnosis of GAD, primary diagnosis of SP, and family income, see Table 1). For the TPOCS-A model, the intercept differences between ICBT and YAS-UC (γ = .54, d = 1.11, p < .001) and YAS-ICBT and YAS-UC (γ = .38, d = .78, p = .002) remained statistically significant. However, the difference between the two ICBT groups (γ = .16, d = .29, p = .23) was no longer statistically significant. For TPOCS-A slope, results were unchanged (YAS-ICBT vs. ICBT γ = .006, d = .29, p = .34; ICBT vs. YAS-UC γ = −005, d = −.25, p = .371; YAS-ICBT vs. YAS-UC γ = −01, d = −54, p = .012). For the TASC, controlling for youth characteristics also did not change the results (ICBT vs. YAS-UC γ = 3.43, d = .84, p = .036; ICBT vs. YAS-ICBT γ = 5.19, d = 1.27, p = .002; YAS-ICBT vs. YAS-UC γ = −1.76, d = −.43, p = .24).

To understand which youth characteristics might explain the intercept differences between ICBT and YAS-ICBT, we conducted post-hoc analyses entering each youth characteristic separately into the TPOCS-A models. These analyses indicated that the two ICBT groups were no longer statistically significantly different from one another when primary phobia diagnosis (γ = .11, d = .23, p = .253), primary GAD diagnosis (γ = .16, d = .33, p = .121), and CBCL Anxious-Depressed score (γ = .19, d = .39, p = .082) were entered as control variables. Thus, differences in primary anxiety diagnoses and symptom severity seemed to be drivers of alliance differences between the two ICBT groups at the first session.

Discussion

Does the strength of the alliance differ for an EBT delivered in research versus practice settings? We found that the observer-rated and youth-rated alliance was significantly higher for youth who received ICBT in a research setting than for youth who received either ICBT or UC in practice settings. The only statistically significant difference between the treatments delivered in the practice settings was early in treatment, with the observer-rated alliance significantly higher in ICBT than UC. At no point was the alliance lower in either manualized ICBT group than UC. These findings indicate that, in the samples used here, employing a manual-based ICBT program did not interfere with alliance formation, but alliance may be stronger in research settings.

This is the first study to evaluate whether the alliance varies when the same EBT is delivered across settings. Consistent with previous research using observer-rated alliance instruments (e.g., Hudson et al., 2014; Liber et al., 2010), each group displayed a relatively flat alliance trajectory. This suggests that across each group the alliance formed early and was relatively unaffected by what happened during treatment. Thus, it is important to consider what factors may have caused initial alliance scores to vary across these settings.

Starting with youth-level factors, it is possible that some youth may present to therapy better equipped to form a strong alliance than others (Baldwin, Wampold, & Imel, 2007; DeRubeis, Brotman, & Gibbons, 2005). Consistent with prior studies (e.g., Ehrenreich et al., 2011; Southam-Gerow et al., 2008), in our sample there were demographic and clinical differences between youth participants in the research and practice settings. When these variables were controlled for, a few group differences were no longer statistically significant. Specifically, when differences in clinical characteristics were controlled (i.e., CBCL Anxious-Depressed score, primary phobia diagnosis, primary GAD diagnosis), the difference between the two ICBT groups in observer-rated alliance was no longer statistically significant, though the ICBT groups remained significantly higher than UC. Previous research has found that levels of pre-treatment symptomatology are related to the quality of the alliance (e.g., Chu et al., 2014; LoTempio et al., 2013), though the direction of this relationship has been inconsistent. That said, it is possible that the differences in youth clinical presentations between settings may have, in part, influenced client-therapist alliance.

Though we assessed whether a number of youth characteristics accounted for the alliance differences, it is possible that youth factors not measured in this study explain our findings. In previous research, other youth factors (e.g., social competencies, coping style, number of services received) have been associated with the strength of the alliance (e.g., Chu et al., 2014; Cummings et al., 2013; Kazdin & Durbin, 2012). Though not measured here, such factors may help explain the lower alliance scores in the practice settings. It is also possible that differences in youth treatment expectations and motivation level may have contributed to the dissimilar alliance levels across settings. Youth rarely refer themselves to treatment (Armbruster & Kazdin, 1994; Ollendick & Vasey, 1999) and typically do not agree with adults (caregiver, therapist) about the goals for treatment (Hawley & Weisz, 2003; Yeh & Weisz, 2001) or even the problems that should be targeted (Weisz et al., 2011). The two groups relied on different referral streams: the research sample was recruited whereas the practice sample was clinic-referred and thus did not hear of any study until after they got to the clinic. This difference may be associated with potentially important dissimilarities in perspective and motivation for treatment. Given that the families sought out the type of treatment provided in the research setting, it is possible that motivation to change was higher for these clients (i.e., the action stage; Prochaska & Norcross, 2001) and with expectations that align with ICBT for youth anxiety. Relatively little research has investigated the relation between expectations, motivation levels, and alliance formation in youth therapy. However, in adult therapy, lower motivation to change has been linked to alliance problems in CBT for generalized anxiety (Hunter, Button, & Westra, 2014) and expectations for treatment have been linked to higher engagement in therapeutic activities in CBT for anxiety (Westra, Dozois, & Marcus, 2007). Thus, the role of youth expectations and motivation for treatment in alliance is an important future research direction.

Therapist-level factors represent another possible source of variation in the alliance (Baldwin et al., 2007; DeRubeis et al., 2005) as it is possible certain therapists may be able to form a stronger alliance with youth than others. This possibility was supported by our data, in which significant variability in both baseline observed alliance and youth-report post-treatment alliance were at the therapist level. The therapists in the research and practice settings differed in important ways. For example, all therapists in the research setting had a psychology background whereas only 40% of the therapists in the practice settings had a psychology background. Previous research has found that more experienced therapists (Mallinckrodt & Nelson, 1991) and training in a specific treatment model (Hilsenroth, Ackerman, Clemence, Strassle, Handler, 2002) are associated with a stronger alliance. Given that the research setting specialized in the treatment of youth anxiety, it is possible that therapists in this setting received more specialized training in CBT. Thus, variations in training, levels of experience, and/or opportunities provided in the setting may have contributed to the observed alliance differences. However, it is important to highlight that therapist professional background and training was largely confounded with setting in our sample making it difficult to ascertain if there is indeed a relation between training and alliance formation. Given potential differences in how therapists with various backgrounds are trained (e.g., Weissman et al., 2006) and the possible impact of experience, these represent important directions for future research to address.

Another factor that might explain our findings is the use of specific therapeutic interventions. Early in treatment the alliance was significantly higher in both ICBT conditions compared to UC. This raises the possibility that the interventions, structure, and/or collaborative stance of ICBT may help promote alliance formation (e.g., encouraging collaboration, setting common goals; Creed & Kendall, 2005; Diamond et al., 1999). Though UC likely contained some CBT interventions, whatever CBT interventions included in UC were most likely delivered at a lower dose than the CBT interventions found in both ICBT conditions (McLeod & Weisz, 2010; Southam-Gerow et al., 2010; Weisz et al., 2009). Of course, any boost to the alliance conferred by ICBT in the practice settings did not appear to last, as there was no significant difference between ICBT and UC by the end of treatment. This suggests that the interventions and/or structure of ICBT may help promote initial alliance formation in practice settings.

Finally, future research could address the possibility that differences in setting factors may have influenced the quality of the alliance. For example, it is plausible that differences in caseloads, productivity requirements, and/or paperwork could have impacted the alliance via job performance (Green et al., 2014). Recently, Green et al. (2014) also found a relation between organizational variables and the quality of the youth-therapist alliance. Specifically, transformational leadership style was positively related to the alliance and this relationship was mediated by organizational climate. This finding adds to a growing body of studies that suggest organizational variables may represent important facilitators or barriers to treatment success in practice settings (e.g., Glisson & Green, 2006; Glisson & James, 2002). As the alliance represents an indicator of service quality (Green et al., 2014), it is important for future work to try and clarify how organizational factors might influence the alliance in practice settings.

Beyond demonstrating that the alliance may differ across settings, our findings contribute to a growing body of research that suggests that the alliance may change over treatment (see Chu et al., 2014; Hudson et al., 2014; Kendall et al., 2009). In the present study, the observer-rated alliance had a slight negative trajectory, which is consistent with previous research using observer-rated alliance instruments (e.g., Hudson et al., 2014; Liber et al., 2010). However, these findings run counter to studies that indicate self-report alliance improves over time (Chu et al., 2014; Kendall et al., 2009). These contrasting findings raise the possibility that the slight deterioration in alliance observed with observer-rated instruments may be related to a reliance on observational measurement. As noted in our sample (r = .38), self- and observer-report alliance instruments do not evidence substantial overlap (e.g., Fjermestad et al., 2012), suggesting possible room for divergence in findings. Our results do, however, indicate that future studies should include repeated self- and observer-report alliance assessments so that the shape of the alliance over treatment for both methods can be clarified.

Future research directions are indicated by some study limitations. Of primary importance is to replicate our findings across a wider range of “research” and “practice” settings. Though the differences in the research and practice settings noted in our sample are consistent with previous research focused on anxious youth (Ehrenreich et al., 2011; Southam-Gerow et al., 2003), it is critical to make clear that no sample of settings can be completely representative of all research or practice settings. The characteristics of particular research and practice settings will be shaped to an extent by the therapists, youth, and the larger context in which treatment is provided. It thus is important not to assume that our study findings will generalize to all “research” or “practice” settings. Rather, our study serves to highlight that the alliance may differ across settings and help point directions for future research that can help clarify if certain therapist, client, or settings factors contribute to this difference.

Our findings should also be replicated with a larger sample of youth with diverse clinical presentations; our study focused solely on youth with anxiety disorders so it is important to ascertain whether the findings generalize to other internalizing and externalizing disorders. Also our findings only speak to the youth-therapist alliance and omit other important alliance relationships, such as the parent-therapist alliance. The alliance, client involvement, and outcome were not assessed throughout treatment in each study. As conceptual models (Hill, 2005; Shirk & Karver, 2006) and some empirical findings (Marker, Comer, Abramova, & Kendall, 2013; McLeod et al., 2014; Zilcha-Mano, Dinger, McCarthy, & Barber, 2014) suggest that these variables may exhibit a reciprocal relationship over treatment, we may have missed important information that could explain the differences across groups (e.g., early improvements in outcome predicting a stronger alliance; Kazdin, 2007). We used a shortened version of the TASC, which may limit comparisons with studies that used the full version. Moreover, our version of the TASC focused exclusively on the bond dimension of the alliance whereas the TPOCS-A assesses the bond and task alliance dimensions. This means that the findings from the two instruments may not be directly comparable.

As mentioned above, there are a number of factors that could have influenced alliance that were not available in this dataset, such as youth social competencies (Kazdin & Durbin, 2012) or attachment history (Zack et al., 2015), and we were unable to examine important therapist and organizational factors. Moreover, although youth- and therapist-level factors are important to consider separately, it is possible that client-therapist interactions may account for variation in the alliance (Baldwin et al., 2007; DeRubeis et al., 2005). Some competent therapists may be able to form a strong alliance with all youth whereas other therapists may only be able to form a strong alliance with youth who present to treatment with the ability to form a strong alliance. It is possible that a greater range of therapist competencies are required to form an alliance with racially/ethnically diverse youth and/or youth who present with both internalizing and externalizing problems (e.g., ability to deal with hostile versus shy youth; Shirk & Karver, 2003). Thus, the generalist treatment approach utilized in many practice settings may make it more difficult to form a strong alliance with the clinically complex, diverse youth who present to treatment in these settings. As the clinicians in our study typically only treated an average of 1.96 participants (SD = 1.86), it was not possible to examine such interactions in this study, but future research should investigate whether the interaction between therapist competencies and youth heterogeneity accounts for variation in the alliance.

Our study also has certain strengths. By using the same EBT delivered in research and practice settings, we were able to glean information on alliance in the two settings when there were no differences in the specific treatment employed. Accurate measurement of the alliance construct was aided by observer-rated data in addition to youth-report. Finally, investigating differences in the alliance over time proved important and suggest avenues for future research.

The findings have clinical implications. A potential barrier to therapists’ use of EBTs is hesitance to use manual-based treatments due to concerns that they will undermine alliance. Our findings indicate comparable alliance between groups in the practice settings, suggesting this concern may not have merit. In fact, our findings suggest the possibility that the structure and/or content of ICBT may be helpful in the task of early alliance building. Further, our results suggest that the alliance may not be as strong overall for youth in practice settings than research settings. To engage the youth who present to treatment in practice settings in treatment it may be critical for therapists to build a strong alliance early in treatment in order to maximize involvement in therapeutic activities (Shirk & Saiz, 1992), and our findings suggest the possibility that learning to deliver EBTs may also enhance therapists’ success in alliance-building. Future research will need to strive to identify what youth, therapist, setting, or interactive factors contribute most to alliance formation in practice settings. This represents a potential area for future research and quality improvement efforts, as efforts to strengthen the alliance may ultimately improve outcomes in practice settings.

Public Health Significance Statement.

This study suggests that the child-therapist alliance may be stronger in university-based research setting compared to community-based mental health settings. It may be more difficult to form as strong of a child-therapist alliance in community-based mental health settings.