Reliability, Factor Structure, and Validity of an Observer-Rated Alliance Scale with Youth

Abstract

Measurement limitations lessen the conclusions about the role of the alliance in youth psychosocial treatment. This paper examined the score reliability, factor structure, and validity of the 9-item Therapy Process Observational Coding System for Child Psychotherapy – Alliance scale (TPOCS-A). The sample, 51 youth aged 7-15 years (M age = 10.36 years, SD = 1.90; 86.3% White; 60.8% male, 39.2% female), met diagnostic criteria for a principal anxiety disorder and received cognitive-behavioral therapy. Treatment sessions (N = 463) were coded by independent coders using the TPOCS-A along with observational measures of treatment adherence and therapist competence. Youth and therapists also completed self-report alliance measures at the end of each session. Reliability estimates, ICC(2,2), at the item level indicated a mean interrater reliability of .68 (SD = .10) and a mean coder stability of .64 (SD = .11). An exploratory factor analysis identified a one-factor solution with five items. TPOCS-A scores evidenced convergent validity with the therapist- and adolescent-reports of alliance but did not converge with the child-report alliance measure. TPOCS-A scores evidenced discriminant validity when compared with scores on adherence and competence measures. The reliability and validity profile of the 9- and 5-item versions of the TPOCS-A were similar. Overall, findings support the reliability and validity of the TPOCS-A scores, but questions remain about how best to assess the alliance with children.

The alliance, defined as the affective and collaborative components of the client-therapist relationship (Elvins & Green, 2008; Shirk & Saiz, 1992), is considered important to the success of psychosocial treatments for youth and their families (Fjermestad et al., 2016; Karver et al., 2018). Meta-analyses have found that the alliance evidences a consistent, but small association with youth treatment outcomes (rs = .14 to .22; Karver et al., 2018; McLeod, 2011; Shirk et al., 2011). The evidence supporting the alliance as a predictor of positive clinical outcomes led the American Psychological Association to consider the alliance an important component of evidence-based practice (American Psychological Association, 2006).

The past two decades have witnessed an expansion of research focused on the role the alliance plays in the process and outcome of treatments for children and adolescents (hereafter called youth; Karver et al., 2018; McLeod, 2011). However, measurement limitations have slowed research progress (Elvins & Green, 2008; Karver et al., 2018). Variations in how the alliance is defined and measured across studies have undermined drawing conclusions (Elvins & Green, 2008; Karver et al., 2018). For example, in a meta-analysis that focused on the alliance-outcome association, McLeod (2011) found that 16 distinct measures were used in 34 studies to assess the youth-therapist alliance. Recently, based on their meta-analytic review, Karver et al. (2018) concluded that limitations related to the measurement of the alliance, such as how the alliance is defined and the lack of variability on alliance measures, represent a barrier to scientific progress.

To address measurement limitations researchers have recommended that studies evaluate the score reliability and validity of measures used to assess the youth-therapist alliance (e.g., Elvins & Green, 2008; McLeod, 2011). The current study reports on the score reliability and validity of the Therapy Process Observational Coding System for Child Psychotherapy-Alliance scale (TPOCS-A; McLeod & Weisz, 2005) using a sample of youth receiving cognitive-behavioral therapy (CBT) for youth anxiety disorders (Kendall et al., 2008). The TPOCS-A is the most widely used observer-rated measure employed to assess the youth-therapist alliance in both child and adolescent samples (see Karver et al., 2018; McLeod, 2011). Compared to self-report alliance measures, observer-rated measures provide certain advantages for the assessment of the alliance in youth treatment, as a youth’s ability to accurately report on their relationship with a therapist may be limited by developmental factors (Shirk & Saiz, 1992). To date, the TPOCS-A has been used to assess the youth-therapist alliance in play therapy, CBT, usual clinical care (e.g., Boyer et al., 2018; Halfon et al., 2019; McLeod & Weisz, 2005) and with youth seeking treatment for a variety of presenting problems (e.g., autism, internalizing, externalizing; e.g., Boyer et al., 2018; Chiu et al., 2009; Halfon et al., 2019; Liber et al., 2010).

Previous studies of the TPOCS-A demonstrated evidence of interrater reliability across various treatment modalities and youth problem types (Boyer et al., 2018; Halfon et al., 2019; McLeod & Weisz, 2005). In terms of validity, TPOCS-A scores have shown evidence of convergent validity with scores on adolescent- and therapist-report alliance measures in a sample of youth receiving CBT in Norway (e.g., Fjermestad et al., 2012). Scores on the TPOCS-A also demonstrated evidence of discriminant validity with measures designed to assess how much (adherence) and how well (competence) therapeutic techniques specified in a treatment protocol are delivered (i.e., treatment integrity measures; Brown et al., 2018; McLeod et al., 2018; Southam-Gerow et al., 2016). Of note, these three studies contained samples from two randomized, controlled trials focused on CBT for youth anxiety (Kendall et al., 2008; Southam-Gerow et al., 2010); data from the Kendall et al. (2008) study is included in the current study. Finally, TPOCS-A scores have shown evidence of predictive validity with clinical outcomes in samples collected in the US, the Netherlands, and Turkey (e.g., Chiu et al., 2009; Halfon et al., 2019; Liber et al., 2010).

The psychometric evidence for the TPOCS-A is promising, but questions related to certain dimensions of score reliability, factor structure, and validity remain. The present study addressed these gaps by investigating reliability and validity dimensions related to using the TPOCS-A to assess the quality of the alliance over the course of treatment. First, we assessed coder stability for each TPOCS-A item. Coder stability is a reliability dimension that focuses on the extent to which item scores are consistent across independent teams of coders (Bjaastad et al., 2016). Therefore, coder stability assesses the replicability of item scores and provides an item-level reliability estimate that speaks to the consistency of each item.

The present study also evaluated the factor structure of the TPOCS-A items. Items on the TPOCS-A were designed to assess the bond (i.e., affective aspects of the youth-therapist relationship) and task (i.e., collaboration on treatment activities), two components of the alliance identified as important in youth treatment (see Shirk & Saiz, 1992). However, a previous exploratory factor analysis conducted in a sample of youth receiving CBT for anxiety disorders in Norway indicated that the nine TPOCS-A items loaded onto a single factor (Fjermestad et al., 2012). This finding is consistent with previous findings in the youth treatment field that found items from alliance measures load onto a single factor (e.g., DiGiuseppe et al., 1996; Faw et al., 2005; Hogue et al., 2006). However, research from the adult treatment literature has indicated that the factor structure of alliance measures may vary across samples (e.g., Hatcher & Gillaspy, 2006). Therefore, we conducted an exploratory factor analysis with a new, larger sample to reexamine the single factor structure of scores on the TPOCS-A items.

We also evaluated if TPOCS-A scores evidenced stability over the course of treatment (see e.g., Accurso et al., 2013). Some studies suggest that scores on therapist-, parent-, and youth-report alliance measures improve over treatment (e.g., Kendall et al., 2009). Yet, some research with the TPOCS-A suggests that scores may change over treatment (Halfon et al., 2019), whereas other studies suggest that scores remain stable (Langer et al., 2011). However, no studies have evaluated if scores on observer-, therapist-, and youth-report alliance measures are consistent with one another over the course of treatment. Determining if scores on alliance measures from different informants are stable has important conceptual (e.g., alliance repair and rupture; Eubanks et al., 2018) and measurement (e.g., how often to assess the alliance) implications.

Another aim of this study was to assess the construct validity of TPOCS-A scores. We employed a multidimensional model that included assessment of multiple constructs (alliance, treatment integrity) and methods (observer-, youth-, therapist-report; Hogue, 2002; McLeod et al., 2018). A key focus of these analyses was to determine the extent to which scores on the TPOCS-A converged with scores on youth- and therapist-report alliance measures. A persistent question in the youth treatment field is the extent to which observer-, therapist-, and youth-report alliance measures assess the same construct (Elvins & Green, 2008; McLeod et al., 2017). Youth-report measures do not consistently converge with observer- or therapist-report alliance measures (see Fjermestad et al., 2012). The lack of convergence may be due to measurement limitations or developmental factors. Some have raised concerns that youth-report forms are subject to ceiling effects that limit their variability (see Chu et al., 2004; Karver et al., 2018; Kendall, 1994; Kendall et al., 1997), but few studies have addressed this issue empirically. Others assert that children may be limited in their ability to report on their relationship with a therapist due to developmental factors (McLeod & Weisz, 2005; Shirk & Saiz, 1992). Some evidence supports this hypothesis, as previous research with a subset of the current sample coded by an independent set of coders found that youth-report alliance measures demonstrated less convergence with therapist-report alliance measures and two observer-report alliance measures (i.e., the TPOCS-A and the Vanderbilt Therapeutic Alliance Scale Revised Short Form; Shelef & Diamond, 2008) in child samples (age < 12) than in adolescent samples (McLeod et al., 2017). However, it is unknown whether this was due to measurement or developmental factors. We thus evaluated the score validity of the TPOCS-A to determine if it captures the quality of the youth-therapist alliance.

To advance the study of the alliance in youth treatment, the score reliability and validity of alliance measures need to be evaluated. The goals of this study were to evaluate the reliability (coder stability), factor structure, and construct validity of scores on the TPOCS-A. To achieve these goals, treatment sessions were coded using the TPOCS-A in a sample of youth diagnosed with anxiety disorders receiving CBT in a university-based specialty clinic (see Kendall et al., 2008). Sessions were also coded with observational treatment integrity measures designed to assess adherence (i.e., degree to which a therapist follows a treatment protocol) and competence (i.e., how well a therapist delivers treatment; Barber et al., 2007). Youth and therapists reported on the alliance with the Therapeutic Alliance Scale for Children (TASC-C/T; Creed & Kendall, 2005), the most commonly used alliance measure used in samples of children and adolescents (Karver et al., 2018; McLeod, 2011). To evaluate the score validity of the TPOCS-A, we evaluated the magnitude and pattern of correlations among scores on the TPOCS-A, self-report alliance measures, and treatment integrity measures designed to assess adherence and competence.

Method

Participants

The treatment data are from a randomized controlled trial that compared the efficacy of three treatments for youth anxiety disorders: individual-CBT, family-focused CBT, and an active control condition (Kendall et al., 2008). The present study focused only on the youth and therapist participants in the individual-CBT group. To be included, youth participants in the parent study had to meet the following criteria: (a) a minimum of two audible treatment sessions, and (b) CBT delivered by a single therapist.

The 51 youth participants were diagnosed with a principal anxiety disorder (i.e., generalized anxiety disorder, separation anxiety disorder, social phobia) and ranged in age from 7 to 15 (M age = 10.36 years, SD = 1.90; 86.3% White; 60.8% male, 39.2% female; see Table 1). A total of 16 therapists (81.1% White, 6.3% Asian/Pacific Islander, 6.3% Latinx, and 6.3% did not report; 12.5% male, 87.5% female) provided treatment in a clinic that specialized in the treatment of anxiety disorders in youth. The therapists were clinical psychology doctoral trainees or licensed clinical psychologists. At post-treatment, 64.0% of the youths in the individual CBT condition no longer met diagnostic criteria for their principal anxiety disorder. All relevant institutional review boards approved the study procedures.

Table 1.

Youth Descriptive Data

Variable	M (SD) or %
Age	10.36 (1.90)
Sex
Male	60.8
Female	39.2
Race/Ethnicity
White	86.3
Black	9.8
Latinx	2.0
Other	2.0
Child Behavior Checklist
Total scale	62.92 (8.56)
Internalizing scale	67.08 (8.60)
Externalizing scale	52.76 (10.08)
Primary Diagnoses
Generalized anxiety disorder	37.3
Separation anxiety disorder	29.4
Social phobia	33.3
Family Income
Up to 60k per year	35.3
Number of Sessions	15.92 (1.43)
Weeks in Treatment	19.52 (3.97)

Note. N = 51.

Individual Cognitive Behavioral Therapy

Therapists delivered the Coping Cat program, a CBT program designed to address youth anxiety disorders (Kendall & Hedtke, 2006). Coping Cat has two phases; (a) training in anxiety management skills (e.g., changing self-talk, problem-solving) and (b) exposure tasks. Therapists were trained by an expert and provided regular supervision. A treatment integrity check indicated that adherence to the Coping Cat protocol was high (see Kendall et al., 2008).

Observational Alliance Measure

Therapy Process Observational Coding System for Child Psychotherapy-Alliance scale

(TPOCS-A; McLeod & Weisz, 2005). The 9-item TPOCS-A assesses the affective elements of the client–therapist relationship and degree of client participation in therapeutic activities. Each item is rated by coders on a 6-point Likert-style scale ranging from 0 (not at all) to 5 (a great deal). In previous studies using independent samples from the one used herein, the TPOCS-A has demonstrated item interrater reliability ranging from ICC(2,2) = .42 to .93 (Chiu et al., 2009; Fjermestad et al., 2012; Liber et al., 2010), internal consistency ranging from α = .80 to .92 (Chiu et al., 2009; Fjermestad et al., 2012; Liber et al., 2010), convergent validity ranging from r = .50 to .54 (Fjermestad et al., 2012), and predictive validity for clinical outcomes (Chiu et al., 2009; Liber et al., 2010). All available treatment sessions were coded with the TPOCS-A.

Measures for Validity Analyses

Therapeutic Alliance Scale for Children – Child and Therapist Versions

(TASC-C/T; Creed & Kendall, 2005). The 12-item TASC-C is designed to assess the affective bond with the therapist and agreement on therapeutic activities. The TASC-T evaluates the therapist’s perspective of how the youth views these two alliance dimensions (e.g., “This child liked spending time with you”). Items on the TASC-C/T are scored on a 4-point Likert-style scale that ranges from 1 (not true at all) to 4 (very true). Scores on the TASC-C/T items are summed (max score = 48) to create a total score, with higher scores representing a stronger alliance. TASC-C/T scores have demonstrated convergent validity (Accurso & Garland, 2014; Fjermestad et al., 2012) and predictive validity for clinical outcomes (e.g., Accurso et al., 2013; Hawley & Weisz, 2005). The TASC-C and TASC-T were collected following each session.

CBT Adherence Scale for Youth Anxiety

(CBAY-A; Southam-Gerow et al., 2016) is a 22-item observational measure designed to assess adherence to CBT for youth anxiety. The CBAY-A items assess three domains: (a) Standard items that represent common CBT techniques (e.g., Homework Review), (b) Model items that assess model-specific content (e.g., Exposure), and (c) Delivery items that measure how model items are delivered (e.g., Rehearsal). Each item is rated by a coder on a 7-point Likert-style extensiveness scale (Hogue et al., 1996) with the following anchors: 1 (not at all), 3 (somewhat), 5 (considerably), and 7 (extensively). Model items are combined to create a Total scale, and two subscales (Skills, Exposure). Scores on the CBAY-A items, subscales, and scale have demonstrated evidence of construct validity (Cox et al., 2020; McLeod et al., 2018; Southam-Gerow et al., 2016). For the validity analyses in the current study, the Skills (Psychoeducation Anxiety, Emotion Education, Fear Ladder, Relaxation, Cognitive Anxiety, Problem Solving, Self-Reward, Coping Plan) and Exposure (Coping Plan, Exposure Preparation, Exposure, and Exposure Debrief) subscales were used. Each available treatment session was coded with the CBAY-A. The CBAY-A Skills and Exposure subscale scores were produced by taking the highest scoring item from the subscale for each session (see Southam-Gerow et al., 2016). For the present sample, the interrater reliability of scores produced by the coders for the CBAY-A Skills and Exposure subscales were ICC(2,2) = .72 and .87, respectively.

CBT for Youth Anxiety Competence Scale

(CBAY-C; McLeod et al., 2018) is a 25-item observational measure that is designed to assess therapist competence in delivering CBT for youth anxiety. The content of the CBAY-C parallels that of the CBAY-A with items assessing the same three domains: Standard, Model, and Delivery. Ratings are made on a 7-point Likert-type competence scale with the following anchors: 1 (very poor), 3 (acceptable), 5 (good), and 7 (excellent). The CBAY-C item and scale scores have demonstrated evidence of construct validity (Cox et al., 2020; McLeod et al., 2018). We used the CBAY-C Total scale comprised of 11 items (Psychoeducation Anxiety, Emotion Education, Fear Ladder, Relaxation, Cognitive Anxiety, Problem Solving, Self-Reward, Coping Plan, Exposure Preparation, Exposure, and Exposure Debrief) for analyses as the Skills and Exposure subscales were highly correlated (r = .64). All available treatment sessions were coded with the CBAY-C. The CBAY-C Total scale score was produced by taking the highest scoring item from the scale for each session (see McLeod et al., 2018). For the present sample, the CBAY-C Total scale interrater reliability for scores produced by the coders was ICC(2,2) = .61.

Global Therapist Competence Scale for Youth Psychosocial Treatment

(G-COMP; Brown et al., 2018) is an observational measure designed to assess global competence in youth treatment and is comprised of five items that assess domains: Alliance Building (i.e., empathy and demonstrating understanding), Positive Expectancies (i.e., bolstering client’s beliefs about the helping process), Focusing Treatment (i.e., structuring and focusing treatment sessions), Instigating Change (i.e., encouragement of emotional reactions and guided self-exploration), and Responsiveness (i.e., handling resistance and tailoring treatment). Coders produced competence ratings on each item using a 7-point Likert-style scale with the following anchors: 1 (very poor), 3 (acceptable), 5 (good), and 7 (excellent) (Brown et al., 2018). Scores on the G-COMP have demonstrated evidence of interrater reliability (ICC[2,2]s that ranged from .61 to .79 (M = .70, SD = .07) and construct validity (Brown et al., 2018). Scores on the G-COMP were produced by averaging the item scores (see Brown et al., 2018). All available treatment sessions were coded with the G-COMP. In the present sample, the interrater reliability of the scored generated by the coders for the G-COMP scale was ICC(2,2) = .71.

Assessments Collected in the Original RCTs

Diagnostic and symptom measures collected in the original randomized controlled trial were used for descriptive purposes. The Anxiety Disorders Interview Schedule for DSM-IV: Child and Parent Versions (ADIS-C/P; Silverman & Albano, 1996) was used to assess youth DSM-IV disorders. The Child Behavior Checklist (CBCL; Achenbach, 1991) was used to assess symptoms via T-scores on three broad-band scales: Total, Internalizing, and Externalizing.

Procedures for the Alliance and Treatment Integrity Measures

Coding procedures: Main sample.

Two clinical psychology doctoral students (M age = 24.00 years, SD = 1.41; 100.0% female; 50.0% White, 50.0% Asian American) rated the TPOCS-A, four clinical psychology doctoral students (25.0% male, 75.0% female; M age = 28.00, SD = 2.71; 50.0% White, 50.0% Latinx) rated the CBAY-C, and two clinical psychology doctoral students (M age = 27.00 years, SD = 4.23, 50.0% male, 50.0% female; 50.0% White, 50.0% Latinx) rated the CBAY-A. Coders on the three teams were trained over a 3-month period, followed by weekly meetings to prevent coder drift (Margolin et al., 1998). Sessions were randomly assigned to coders from each team, and every session was double-coded to assess interrater reliability. For each measure, a mean score for each item (i.e., average of coder 1 and coder 2’s ratings) was used in analyses to reduce measurement error. All sessions for each case were coded except for the first and last sessions, as these sessions may contain intake or termination content. Coders generated scores on the TPOCS-A if the youth was present for at least 10 minutes of the session, which meant that 463 of the 532 available treatment sessions were used in the present study.

Coding procedures: Coder stability sample.

The sample used for the coder stability analyses represented scores produced by an independent coding team used (see McLeod et al., 2017). Three doctoral students (M age = 26.00 years, SD = 2.00; 100.0% female; 66.0% White, 33.0% Asian-American) comprised the coding team for the TPOCS-A used for the stability sample. The same procedures described above for coder training and session assignment were used. Two sessions were selected from each case for coding: session 3 and 12. When a session was not available, it was replaced with another from the same treatment phase: early (sessions 2-8) or late (sessions 9-16). A total of 99 sessions from the total sample of sessions that the youth was present for more than 10 minutes (N = 463) were coded for stability analyses. See McLeod et al., 2017 for a full description of the coding and sampling procedures.

Procedures: Self-report alliance measures:

The youth and therapist filled out the TASC-C/T at the end of each session. Research staff asked the youth fill out the form and place it in a sealed box. Therapists and youth were naïve to all ratings (see Kendall et al., 2009).

Data Analytic Strategy

Our analyses progressed through four steps designed to evaluate the reliability, representative validity, and elaborative validity of the TPOCS-A scores.

Interrater reliability, item performance, and coder stability.

Interrater reliability was calculated using the two-way random intraclass correlation coefficient (ICC[2,2]; Shrout & Fleiss, 1979). Cicchetti (1994)’s guidelines were used to evaluate ICCs; ICCs below .40 reflected “poor” agreement, ICCs from .40 to .59 reflected “fair” agreement, ICCs from .60 to .74 reflected “good” agreement, and ICCs of .75 and higher reflected “excellent” agreement. Descriptive statistics were based on the mean item score produced by averaging across coders for each item. Coder stability was evaluated using ICC(2,2), by comparing the mean score produced by each coding team for each item.

Exploratory factor analysis.

An exploratory factor analysis with direct oblimin rotation evaluated the factor structure of the nine TPOCS-A items. The sandwich estimator that adjusts standard error estimates for clustering was used to account for the nesting of sessions within youth participants (Asparouhov, 2005). After the initial extraction, items with standardized loadings lower than .40 (indicative of poor fit with the larger construct; Tabachnick et al., 2007) were removed, and the process was repeated until a final factor solution was identified. To produce a final factor solution, both eigenvalue decline and interpretability were considered. Once a final solution was derived, the internal consistency of the factor solution was calculated using Cronbach’s alpha.

Stability.

To account for nesting (i.e., youth in therapists, repeated measures in youth), multilevel modeling (Raudenbush & Bryk, 2002) using HLM 7.01 (Raudenbush et al., 2011) was used to evaluate if scores on the TPOCS-A, TASC-T, or TASC-C changed over the course of treatment. A model was fit for each measure that included a linear term and a quadratic term to test for both possible patterns of change.

Construct validity.

We evaluated the construct validity of scores on the TPOCS-A by assessing the magnitude of the correlations between the TPOCS-A, self-report alliance measures (TASC-C/T), an observational measure of adherence (CBAY-A), and two observational measures of therapist competence (G-COMP, CBAY-C). The correlations were interpreted following Rosenthal and Rosnow’s (1984) guidelines: r was “small” if r = .10 to .23, “medium” if r = .24 to .36, and “large” if r > .36. For follow-up contrasts, contrasts were calculated from the correlation coefficients taking into consideration the fact that the correlations were not independent (Meng et al., 1992).

Results

Interrater Reliability, Item Performance, and Coder Stability

ICC(2,2)s for the nine TPOCS-A items ranged from .52 to .82 (M = .68, SD = .10; see Table 2). The ICC(2,2)s for two of the nine items fell within the “excellent” range, five within the “good” range, and two within the “fair” range (Cicchetti, 1994). All nine items displayed nearly the full range of scores. These findings suggest that the TPOCS-A items demonstrated adequate interrater reliability.

Table 2.

Descriptive Data, Interrater Reliability, Coder Stability, and Factor Loadings for TPOCS-A Items and Scales

TPOCS-A Item	M	SD	Range	Skewness	Kurtosis	Interrater Reliability ICC(2,2)	Coder Stability ICC(2,2)	Factor Loadings	Alpha
1. Feeling Understood	1.23	0.90	4.00	0.38	−0.38	.61	.44	0.61
2. Hostile Manner	4.88	0.46	3.50	−4.75	24.60	.52	.72
3. Positive Affect	1.55	1.06	5.00	0.40	−0.30	.70	.69	0.70
4. Shares Experiences	3.07	0.82	5.00	−1.19	2.40	.74	.77	0.74
5. Client Discomfort	4.69	0.77	5.00	−3.00	9.42	.82	.78	0.80
6. Client/Therapist Discomfort	4.90	0.35	2.50	−3.96	16.64	.53	.56
7. Changes Outside	2.19	1.18	4.50	−0.75	−.70	.74	.64
8. Client Noncompliance	4.86	0.54	4.00	−4.87	25.46	.76	.54
9. Work Equally	3.17	0.75	5.00	−1.23	2.95	.70	.64	0.78
TPOCS-A Scale	3.39	0.52	3.61	−1.50	3.61	.833	.79		.82
TPOCS-AB Scale	2.74	0.69	4.40	−0.95	−0.95	.827	.78		.86

Note. TPOCS-A = The Therapy Process Observational Coding System for Child Psychotherapy-Alliance scale; TPOCS-AB = TPOCS-A Brief scale; ICC = intraclass correlation coefficient.

For the coder stability analyses, the ICC(2,2)s for the nine TPOCS-A items ranged from .44 to .78 (M = .64, SD = .11; see Table 2). The stability estimates for two of the items fell within the “excellent” range, four within the “good” range, and three within the “fair” range (Cicchetti, 1994). This set of analyses indicated that six of the TPOCS-A items demonstrated at least good stability across the two independent coding teams.

Though all items were considered to be at least “fair” interrater reliability (ICC > .40; Cicchetti, 1994), three items had a kurtosis over 10 (Hostile Manner, Client/Therapist Discomfort, Client Noncompliance; see Table 2). Due to the extreme kurtosis values, we removed these three items from subsequent analyses.

Exploratory Factor Analysis

With the three items dropped due to extreme kurtosis values, in our initial analysis of the six TPOCS-A items we extracted two factors. This analysis yielded only a single factor with an eigenvalue greater than 1.0 (eigenvalue = 3.35) that accounted for 55.9% of the variance. A visual inspection of the scree plot showed a substantial drop from the first to the second factor, followed by stabilization. One item, Changes Outside of Session, did not load above .40 on either a one- or two-factor solution, so we reran the factor solution with this item dropped. This new analysis with five TPOCS-A items again produced only one factor with an eigenvalue greater than 1.0 (eigenvalue = 3.26) that accounted for 65.2% of the variance. Moreover, a visual inspection of the scree plot indicated a substantial drop from the first to the second factor, followed by stabilization. One- and two-factor solutions were again extracted. In the two-factor solution, the factors evidenced a large correlation (r = .66) and over half of the scale variance was accounted for by Factor 1 (65.2% vs. 15.5%). Two items had loadings above .40 on Factor 1 and three items had loadings above .40 on Factor 2. No items had crossloadings over .30. In contrast, the one-factor solution generated adequate item loadings ranging from .63 to .88, suggesting that each item was representative of the alliance construct (Costello & Osborne, 2005). Considering both the eigenvalue decline and interpretability for the one- and two-factor solutions, the one-factor solution was deemed most appropriate. Specifically, the scree plot indicated a substantial decrease between Factor 1 and Factor 2, Factor 1 accounted for over half of the variance, Factor 2 only explained a modest amount of variance, and the one-factor solution is consistent with previous research (see Fjermestad et al., 2012). We therefore concluded that our factor analytic findings indicate that the five items on the TPOCS-A is best characterized by a one-factor solution. Cronbach’s alpha was .82 for the full 9-item scale and .86 for the 5-item scale (see Table 2).

Stability

To facilitate comparisons between the two versions of the TPOCS-A, we generated scores based on the 9-item version (TPOCS-A) and the 5-item version (TPOCS-A Brief [TPOCS-AB]). Models examining change over time in both of these variables, the TASC-C, and the TASC-T are presented in Table 3. For all variables, there was no significant change over time, suggesting alliance scores remain stable over the course of treatment across observer-, therapist-, and youth-report.

Table 3.

Stability of the Scores on the TPOCS-A, TPOCS-AB, TASC-T, and TASC-C

	Intercept (Alliance at Session 1)			Linear Change			Quadratic change
	Coefficient	S.E.	p	Coefficient	S.E.	p	Coefficient	S.E.	p
TPOCS-A	3.43	.073	<.001	.0038	.0091	.68	−.00042	.00043	.34
TPOCS-AB	2.80	.10	<.001	.0012	.013	.93	−.00034	.00060	.57
TASC-T	39.09	1.55	<.001	.086	.18	.59	.00070	.0092	.93
TASC-C	40.02	1.33	<.001	.23	.17	.20	−.051	.0066	.45

Note. TPOCS-A = The Therapy Process Observational Coding System for Child Psychotherapy-Alliance scale; TPOCS-AB = TPOCS-A Brief scale; TASC-T = Therapeutic Alliance Scale for Children – Therapist Version; TASC-C = Therapeutic Alliance Scale for Children – Child Version; S.E. = standard error.

Construct Validity

Correlations among scores on the alliance (TPOCS-A, TPOCS-AB, TASC-T, TASC-C), competence (CBAY-C, G-COMP), and adherence (CBAY-A) measures can be seen in Table 4. The magnitude of the correlations between the measures ranged from “small” to “large” (Rosenthal & Rosnow, 1984), with the strongest correlations observed between the two versions of the TPOCS-A (r = .94).

Table 4.

Intercorrelations of the Alliance, Competence, and Adherence Measures

	M (SD)	2	3	4	5	6	7	8	9	10	11	12
1. TPOCS-A	3.39 (.52)	.94**	.60**	.27**	.42**	.19**	.13**	.35**	.03	.24**	.31**	.15
2. TPOCS-AB	2.74 (.69)	–	.57**	.31**	.40**	.16**	.08	.30**	.06	.18**	.24**	.11
3. TASC-T	39.06 (8.13)		–	.39**	.40**	.22**	.16**	.33**	.06	.17**	.28**	.19
4. TASC-C	41.64 (8.02)			–	−.01	−.08	−.12*	.02	−.12*	.04	.01	.04
5. G-COMP–Alliance	4.74 (.72)				–	.47**	.33**	.45**	.50**	.19**	.15**	.17
6. G-COMP–Positive	3.89 (.79)					–	.47**	.47**	.38**	.18**	.23**	.18
7. G-COMP–Focusing	4.98 (.87)						–	.68**	.32**	.37**	41**	.18
8. G-COMP–Instigat.	4.92 (.92)							–	41**	41**	.43**	.30
9. G-COMP–Respon.	4.59 (.76)								–	.11*	.03	.15
10. CBAY-C	5.44 (.88)									–	.55**	.17
11. CBAY-A Skills	4.35 (1.70)										–	−.20
12. CBAY-A Exposure	3.69 (1.87)											–

Note. N = 463. The Therapy Process Observational Coding System for Child Psychotherapy – Alliance scale; TPOCS-AB = TPOCS-A Brief scale; TASC-T = Therapeutic Alliance Scale for Children – Therapist Version; TASC-C = Therapeutic Alliance Scale for Children – Child Version; G-COMP = Global Therapist Competence Scale for Youth Psychosocial Treatment; G-COMP–Positive = G-COMP Positive Expectancies scale; G-COMP–Focusing = G-COMP Focusing Treatment scale; G-COMP–Instigat. = G-COMP Instigating Change scale; G-COMP–Respon. =G-COMP Responsiveness scale; CBAY-C = Cognitive-Behavioral Treatment for Anxiety in Youth Competence Scale; CBAY-A = Cognitive Behavioral Therapy Adherence Scale for Youth Anxiety.

^*p < .05

^**p < .01

The magnitude of the correlations between scores on the TPOCS-A and the TASC-T, TASC-C, G-COMP items, CBAY-C Total scale, and CBAY-A subscales ranged from “small” to “large” (Rosenthal & Rosnow, 1984). The strength of the correlations between scores on the TPOCS-A and the (a) self-report alliance measures (TASC-T, TASC-C) ranged from .27 to .60 (M r = .44, SD = .23); (b) competence scales (Alliance Building, Positive Expectancies, Focusing Treatment, Instigating Change, Responsiveness, CBAY-C Total) ranged from .03 to .42 (M r = .23, SD = .14); and (c) adherence subscales (CBAY-A Skills, CBAY-A Exposure) ranged from .15 to .31 (M r = .23; SD = .11). Comparisons indicated that the correlation between the TPOCS-A and the TASC-T (r = .60) was significantly higher than (a) the correlation between the TPOCS-A and the TASC-C (r = .27; z = 7.63, p < .001); (b) the mean of the correlations between the TPOCS-A and the CBAY-C scales (M r = .23; z = 7.56, p < .001); and (c) the mean of the correlations between the TPOCS-A and the CBAY-A subscales (M r = .23; z = 6.65, p < .001). In contrast, the magnitude of the correlation between the TPOCS-A and the TASC-C (r = .27) was not significantly different than (a) the mean of the correlations between the TPOCS-A and the competence scales (M r = .23; z = .66, p = .51), or (b) the mean of the correlations between the TPOCS-A and the CBAY-A subscales (M r = .23; z = .65, p = .52). Overall, the magnitude and pattern of correlations support the construct validity of the TPOCS-A, with the exception of the correlations between the TPOCS-A and the TASC-C, which do not support the convergent validity of the two measures.

The magnitude of the correlations observed between scores on the TPOCS-AB and the TASC-T, TASC-C, G-COMP items, CBAY-C Total scale, and CBAY-A subscales ranged from “small” to “large” (Rosenthal & Rosnow, 1984). The magnitude of the correlations between scores on the TPOCS-AB and the (a) self-report alliance measures ranged from .31 to .57 (M r = .44, SD = .18), (b) competence scales ranged from .06 to .40 (M r = .20, SD = .13), and (c) adherence subscales ranged from .11 to .24 (M r = .18, SD = .09). Comparisons revealed that the correlation between the TPOCS-AB and the TASC-T (r = .57) was significantly higher than (a) the correlation between the TPOCS-AB and the TASC-C (r = .31; z = 5.96, p < .001), (b) the mean of the correlations between the TPOCS-AB and the competence scales (M r = .20; z = 7.38, p < .001); and (c) the mean of the correlations between the TPOCS-AB and the CBAY-A subscales (M r = .18; z = 7.85, p < .001). The magnitude of the correlation between the TPOCS-A and the TASC-C (r = .31) was not significantly different than the mean of the correlations between the TPOCS-A and the competence scales (M r = .20; z = 1.82, p = .07), but was significantly higher that the correlation between the TPOCS-AB and the CBAY-A subscales (M r = .18; z = 2.09, p = .04). Overall, the magnitude and pattern of the correlations between the TPOCS-AB and the remaining measures were similar to those observed for the TPOCS-A.

Examining Alternative Explanations

Our findings thus far raise questions about the convergent validity of the observer-rated measures (TPOCS-A, TPOCS-AB) with the TASC-C. As noted earlier, previous research conducted with a subset of treatment sessions from this sample coded by an independent set of coders indicated that the TASC-C did not converge with other alliance measures in child samples but did converge in a sample of adolescents (McLeod et al., 2017). We thus investigated whether the pattern of correlations among the TPOCS-A, TPOCS-AB, and TASC-C differed across children and adolescents. For the child sample (see Table 5), the correlation between the TPOCS-A and the TASC-C (r = .07) was significantly lower than the correlation between the TPOCS-A and the (a) TASC-T (r = .56; z = 7.60, p < .001), (b) mean of the competence scales (M r = .36, SD = .14; z = 3.79, p < .001), and (b) mean of the CBAY-A subscales (M r = .27, SD = .13; z = 2.51, p < .001). A similar pattern was observed for the TPOCS-AB. The correlation between the TPOCS-AB and the TASC-C (r = .11) was significantly lower than the correlation between the TPOCS-AB and the (a) TASC-T (r = .50; z = 5.90, p < .001), and (b) mean of the competence scales (M r = .31, SD = .11; z = 2.59, p < 001). However, there was no significant difference between the TPOCS-AB and TASC-C correlation and the TPOCS-AB and mean of the CBAY-A subscales correlations (M r = .23, SD = .12; z = 1.50, p = .13).

Table 5.

Intercorrelations of the Alliance, Competence, and Adherence Measures for Children

	M (SD)	2	3	4	5	6	7	8	9	10	11	12
1. TPOCS-A	3.41 (50)	.93**	.56**	.07	.50**	.35**	.36**	.48**	.12*	.36**	.36**	.17**
2. TPOCS-AB	2.78 (.64)	–	.50**	.11	.46**	.29**	.30**	.42**	.14	.27**	.31**	.14*
3. TASC-T	39.15 (7.71)		–	.22**	.42**	.32**	.28**	.35**	.08	.22**	.23**	.17**
4. TASC-C	41.84 (8.30)			–	−.01	−.08	−.11*	.02	−.12*	.04	.01	.04
5. G-COMP–Alliance	4.76 (.77)				–	.59**	44**	.53**	.56**	.26**	.16**	.16
6. G-COMP–Positive	3.87 (.82)					–	.54**	.57**	.46**	.24**	.20**	.21**
7. G-COMP–Focusing	4.94 (.86)						–	.73**	.33**	.34**	.35**	.23
8. G-COMP–Instigat.	4.92 (.95)							–	.44**	.43**	.29**	.32
9. G-COMP–Respon.	4.61 (.80)								–	.16**	.08	.12*
10. CBAY-C	5.47 (.82)									–	.37**	.15*
11. CBAY-A Skills	4.32 (1.74)										–	−.17**
12. CBAY-A Exposure	3.66 (1.88)											–

Note. N = 294. The Therapy Process Observational Coding System for Child Psychotherapy – Alliance scale; TPOCS-AB = TPOCS-A Brief scale; TASC-T = Therapeutic Alliance Scale for Children – Therapist Version; TASC-C = Therapeutic Alliance Scale for Children – Child Version; G-COMP = Global Therapist Competence Scale for Youth Psychosocial Treatment; G-COMP–Positive = G-COMP Positive Expectancies scale; G-COMP–Focusing = G-COMP Focusing Treatment scale; G-COMP–Instigat. = G-COMP Instigating Change scale; G-COMP–Respon. =G-COMP Responsiveness scale; CBAY-C = Cognitive-Behavioral Treatment for Anxiety in Youth Competence Scale; CBAY-A = Cognitive Behavioral Therapy Adherence Scale for Youth Anxiety.

^*p < .05

^**p < .01

For the adolescent sample (see Table 6), the correlation between the TPOCS-A and the TASC-C (r = 62) was not significantly different than the correlation between the TPOCS-A and the TASC-T (r = .65; z = .68, p = .50). Moreover, the correlation between the TPOCS-A and the TASC-C (r = .62) was significantly higher than the mean of the correlations between the TPOCS-A and (a) the CBAY-C scale (M r = .15, SD = .07; z = 5.87, p < .001), and (b) the CBAY-A subscales (M r = .07, SD = .06; z = 6.69, p < .001). The same pattern was observed for the TPOCS-AB. The TPOCS-AB and the TASC-C correlation (r = .64) was not significantly different from the correlation between the TPOCS-AB and the TASC-T (r = .67; z = .70, p = .48). Yet the correlation between the TPOCS-A and the TASC-C (r = .64) was significantly higher than the mean of the correlations between the TPOCS-A and the (a) CBAY-C scale (M r = .14, SD = .11; z = 5.82, p < .001), and (b) CBAY-A subscales (M r = .02, SD = .01; z = 6.95, p < .001). Overall, the magnitude and pattern of the correlations between the TPOCS-AB and the remaining measures were similar to those observed for the TPOCS-A. This suggests that the TPOCS-A and TPOCS-AB show adequate convergent validity with adolescent-report alliance measures but not with child-report alliance measures.

Table 6.

Intercorrelations of the Alliance Competence and Adherence Measures for Adolescents

	M (SD)	2	3	4	5	6	7	8	9	10	11	12
1. TPOCS-A	3.36 (.55)	.95**	.65**	.62**	.27**	−.08	−.20**	.14	−.14	.09	−.03	.11
2. TPOCS-AB	2.69 (.78)	–	.67**	.64**	.33**	−.05	−.23	.13	−.06	.06	−.03	.01
3. TASC-T	38.89 (8.86)		–	.70**	.36**	.05**	−.01	.31**	.03	.11	.03	.24**
4. TASC-C	41.28 (7.49)			–	.15	−.06	−.14*	.16	−.04	.12	.00	.14
5. G-COMP–Alliance	4.72 (.64)				–	.21**	.12**	.26**	.37**	.06	−.09	.20**
6. G-COMP–Positive	3.93 (.73)					–	.34**	.25**	.22**	.11	.06	.13
7. G-COMP–Focusing	5.07 (.88)						–	.58**	.31**	.42**	.20**	.11
8. G-COMP–Instigat.	4.93 (.87)							–	.35**	.37**	.08	.26**
9. G-COMP–Respon.	4.55 (.69)								–	.01*	−.14	.21**
10. CBAY-C	5.38 (.98)									–	.37**	.20*
11. CBAY-A Skills	4.41 (1.64)										–	−.26**
12. CBAY-A Exposure	3.74 (1.86)											–

Note. N = 169. The Therapy Process Observational Coding System for Child Psychotherapy – Alliance scale; TPOCS-AB = TPOCS-A Brief scale; TASC-T = Therapeutic Alliance Scale for Children – Therapist Version; TASC-C = Therapeutic Alliance Scale for Children – Child Version; G-COMP = Global Therapist Competence Scale for Youth Psychosocial Treatment; G-COMP–Positive = G-COMP Positive Expectancies scale; G-COMP–Focusing = G-COMP Focusing Treatment scale; G-COMP–Instigat. = G-COMP Instigating Change scale; G-COMP–Respon. =G-COMP Responsiveness scale; CBAY-C = Cognitive-Behavioral Treatment for Anxiety in Youth Competence Scale; CBAY-A = Cognitive Behavioral Therapy Adherence Scale for Youth Anxiety.

^*p < .05

^**p < .01

Our final set of analyses focused on whether ceiling effects may explain the pattern of correlations observed in the child and adolescent samples. In the child sample, 14.8% of the responses were at the maximum on the TASC-T (Skewness = −.71; Kurtosis = −.06), whereas 32.2% of the responses on the TASC-C (Skewness = −1.50; Kurtosis = 1.34) were at the maximum. In the adolescent sample, 8.3% of the responses were at the maximum on the TASC-T (Skewness = −1.26; Kurtosis = 1.02), whereas 14.2% of the responses on the TASC-C (Skewness = −1.62; Kurtosis = 2.42) were at the maximum. None of the scores on the TPOCS-A or TPOCS-AB reached the maximum response in the child or adolescent sample. Guidelines suggest that ceiling effects may be present if more than 15% of the responses achieve the maximum score (Terwee et al., 2007), indicating that ceiling effects may be present for the TASC-C in the child sample.

Discussion

In an examination of the score reliability and validity of an observational measure estimating the alliance, we found that the TPOCS-A items showed adequate interrater reliability and coder stability. Examination of the factor structure supported a one factor solution with five items. Scores on the TPOCS-A, TASC-T, and TASC-C did not systematically change over the course of treatment. The evaluation of the correlations between the TPOCS-A, therapist-report alliance measure, competence measures, and adherence measure supported the construct validity of scores on the 9- and 5-item versions of the TPOCS-A. The magnitude and pattern of correlations among the four alliance measures differed for children and adolescents, which raises concerns about the score validity of observer- and self-report alliance measures used with children.

Within the current sample of youth receiving CBT for anxiety problems, the nine TPOCS-A items evidenced adequate interrater reliability and coder stability. These findings indicate that the coders within a team as well as two independent teams of coders were able to produce scores with adequate reliability at the item level. The item-level interrater reliability is consistent with previous samples (Burnham Riosa et al., 2019; Chiu et al., 2009; Fjermestad et al., 2012; McLeod & Weisz, 2005) and support that item-level scores were reliable and stable.

At the scale level, interrater reliability, coder stability, and Cronbach’s alpha were adequate, using Cicchetti’s (1994) standards, for the 9- and 5-item versions of the TPOCS-A. The interrater reliability and alpha values seen here are consistent with previous research (Burnham Riosa et al., 2019; Chiu et al., 2009; Fjermestad et al., 2012; McLeod & Weisz, 2005). Interrater reliability for the two versions of the TPOCS-A scales were both considered “excellent” (Cicchetti, 1994). Hence, the current findings support the reliability of both the 5-item and 9-item versions of the TPOCS-A.

The present findings indicate that measurement of alliance in youth treatment as a unidimensional construct may be optimal. This finding is consistent with (a) previous research with the TPOCS-A conducted in an independent sample of youth receiving CBT for anxiety (Fjermestad et al., 2012), (b) factor analytic findings for other observer-rated alliance measures in youth treatment (Faw et al., 2005; Hogue et al., 2006), and (c) between-person factor analytic solutions for self-report alliance measures in youth treatment (Accurso et al., 2013; Cirasola et al., 2020). That a group of studies found observer- and self-report alliance measures in youth treatment may have a unidimensional structure suggests that youth may not distinguish between the various components of the alliance. This distinction is important to examine further, as it suggests that youth who establish a strong affective connection with their therapist may be more likely to participate in therapeutic activities and vice versa (Fjermestad et al., 2012).

Though this study supports a unidimensional solution, more research is needed about the structure of the alliance in youth treatment. Karver et al. (2018) noted that many existing alliance measures rely on old definitions of the alliance. The TPOCS-A items were developed from existing measures (see Elvins & Green, 2008; McLeod & Weisz, 2005), and based on a conceptualization of the alliance that emphasizes the importance of the bond and task dimensions in youth treatment (see Shirk & Saiz, 1992). Some have suggested that the definition of the alliance in youth treatment needs to be updated and informed by an understanding of how relationships form for children and adolescents (Karver et al., 2018). Revisiting how the alliance is defined in youth treatment may lead to the development of alliance measures that can distinguish between various components of the alliance.

The results of this study indicate that the scores on the observer-, therapist-, and youth-report alliance measures did not systematically change over the course of treatment. As noted earlier, the extent to which scores on alliance measures change over youth treatment has varied across past studies. To our knowledge, this is the first study to examine if scores on alliance measures collected at each session from observer-, therapist, and youth-report alliance measures vary over treatment. The three different informants in our study provided consistent data regarding the alliance over the course of treatment. Our findings therefore suggest that findings regarding change over treatment from observer-report alliance measures may generalize to self-report alliance measures, and vice versa.

Overall, our findings support the convergent and discriminant validity of scores on both the 5- and 9-item versions of the TPOCS-A. Both versions of the TPOCS-A evidenced stronger correlations with a therapist-report alliance measure and significantly lower correlations with competence and adherence measures. The magnitude and pattern of these correlations support the construct validity of scores on both versions of the TPOCS-A, as these findings are consistent with previous research that has investigated relations between the alliance, competence, and adherence in youth (Fjermestad et al., 2012; Hogue et al., 2008) and adult (e.g., Carroll et al., 2000) treatment. The therapist-report of the alliance (TASC-T) evidenced strong correlations with both versions of the TPOCS-A scales. Correlations between the TPOCS-A and TASC-T were higher than correlations previously found between observer- and therapist-report alliance measures in adult treatment samples (rs from −.22 to .43; Cecero et al., 2001; Tichenor & Hill, 1989). Overall, these findings support the convergent and discriminant validity of scores on the TPOCS-A.

Despite that overall positive picture, our findings raise concerns about the validity of assessing the youth-therapist alliance in child (vs. adolescent) samples. In our adolescent sample (ages 12 to 15 years), correlation coefficients between the TPOCS-A and youth-report TASC-C were “large” (Rosenthal & Rosnow, 1984). In contrast, in our child (ages 7 to 11 years) sample, the correlation coefficients between the TPOCS-A and TASC-C were “small” (Rosenthal & Rosnow, 1984). Our results with children are consistent with correlations reported between client- and observer-rated alliance measures in adult treatment (range from −.33 to .38; Cecero et al., 2001; Tichenor & Hill, 1989). However, the magnitude of the correlation between the TPOCS-A and the TASC-C varied across the adolescent and child samples, which raises questions about the validity of alliance scores in younger samples.

It is not clear why a different pattern of correlations was observed in the child and adolescent samples. The pattern could stem from the origin of the TPOCS-A from a measure developed for adult (vs. child) treatment (Elvins & Green, 2008; McLeod & Weisz, 2005). As noted earlier, developing alliance measures specifically for younger populations may make them more sensitive to how the alliance is formed and maintained in children. In contrast, linguistic or cognitive capabilities may limit a child’s ability to accurately self-report on certain aspects of the alliance (McLeod & Weisz, 2005; Shirk & Saiz, 1992). Measurement issues may also explain the pattern of findings. A critique leveled at youth-report alliance measures is that ceiling effects may limit variability (Chu et al., 2004; Karver et al., 2018; Kendall, 1994; Kendall et al., 1997). Our findings support this claim, as ceiling effects were observed on the TASC-C within the child sample. Our findings thus indicate that there is room for improvement in the measurement of the alliance, particularly with child samples. Until the questions raised by our findings are resolved, it is recommended that studies utilize a multi-method, multi-informant measurement approach to assess the alliance in youth treatment, particularly with child samples (Fjermestad et al., 2012).

Though item-level reliability estimates were adequate, three TPOCS-A items displayed non-normal distributions (Hostile Manner, Client/Therapist Discomfort, Client Noncompliance). The distribution of these items may be explained, in part, by the fact that each focused on negative behavior or interactions. In contrast to positive behaviors and interactions, negative behaviors and interactions have a lower base rate (Creed & Kendall, 2005). This interpretation is consistent with the distribution of each item. A fourth item was dropped as part of the factor analysis, so the final one-factor solution only had five items. This raises the question of what version of the TPOCS-A to use in future research. Reliability and validity findings were very similar across the 9- and 5-item versions of the TPOCS-A, which supports the use of the 5-item version. Additionally, it may be easier to train coders to use the 5-item version. Yet, since most items that were removed focused on negative behavior or interactions, it is possible that the 5-item version may miss important information about the alliance in certain populations (e.g., court ordered populations).

A few limitations bear mentioning. One, the sample was mostly white, relatively affluent (more than 2/3 of families reported incomes over $60K/year), and all with primary anxiety diagnoses. Moreover, they were all receiving CBT provided by expert therapists. As a result, it is possible that the findings may not generalize to other youth populations or usual care treatments.

Considered together, the findings indicate that the observational measure assessing the client-therapist alliance in youth demonstrates promising reliability and validity. These findings add to extant research that suggests that the alliance in youth treatment may be unidimensional. As with previous research, our findings continue to raise concerns about measurement of the alliance in child samples.

Public Significance Statement:

The alliance, defined as the quality of the relationship between youth and therapist, is thought to be an important part of successful youth treatment. The study found that observers can be trained to rate the youth-therapist alliance in youth treatment and that their ratings are comparable to other ways of assessing alliance.