Cognitive vs. Behavioral Skills in CBT for Depressed Adolescents: Disaggregating Within-Patient vs. Between-Patient Effects on Symptom Change

Abstract

Objective:

Despite a growing body of research supporting the efficacy of cognitive behavioral therapy (CBT) for depressed adolescents, few studies have investigated the role of the acquisition and use of CBT skills in accounting for symptom improvement. The present study examined the role of cognitive versus behavioral skills in predicting symptom improvement in depressed youth. Analyses considered different raters of patient skills (patient vs. therapist), as well as disaggregated between-patient versus within-patient effects.

Method:

Data were derived from a 12-week clinical trial of CBT for depressed adolescent females (N = 33; ages 13–18; 69.7% White). Both therapist-report and patient-report measures of CBT skills (Skills of Cognitive Therapy) were acquired at five time points throughout therapy: sessions 1, 3, 6, 9, and 12. Depressive symptoms (Beck Depression Inventory-II) were assessed at every session.

Results:

Therapist and patient ratings of CBT skills showed small to moderate associations (rs =.20 – .38). Intraclass correlation coefficients indicated that the majority of the variance in skills scores (61%−90%) was attributable to within-patient variance from session-to-session, rather than due to between-patient differences. When disaggregating within-patient and between-patient effects, and consistent with a causal relationship, within-patient variability in both patient-rated (b = −2.55; p =.025) and therapist-rated (b = −2.41; p = .033) behavioral skills predicted subsequent symptom change.

Conclusions:

Analyses highlight the importance of the acquisition and use of behavioral skills in CBT for depressed adolescents. Findings also underscore the importance of disentangling within-patient from between-patient effects in future studies, an approach infrequently employed in process-outcome research.

A growing body of research supports the efficacy of cognitive behavioral therapy (CBT) for depressed adolescents (Weersing, Jeffreys, Do, Schwartz, & Bolano, 2017). However, the mechanisms that account for why depressed youth improve—and why many fail to sufficiently improve—in CBT remain unclear (Webb, Auerbach, & DeRubeis, 2012). The cognitive-behavioral model posits that patient acquisition and use of the cognitive and behavioral skills encouraged in treatment (e.g., cognitive restructuring, behavioral activation) contributes to depressive symptom change (Beck, Rush, Shaw, & Emery, 1979). Although there are studies with depressed adults investigating the association between CBT skill use and depressive symptom change (e.g., Hawley et al., 2017; Jarrett et al., 2011, 2013; Strunk, Hollars, Adler, Goldstein, & Braun, 2014; Webb et al., 2016), no study has tested the extent to which depressed adolescents acquire cognitive and behavioral skills in CBT and, critically, whether these skills predict symptom change. Although not focused on treatment, related research on cognitive behavioral depression prevention programs suggests that their beneficial effects may, in part, be mediated by the acquisition of cognitive behavioral skills (Brunwasser, Freres, & Gillham, 2018)

Assuming a statistically significant association between CBT skills and symptom improvement, two additional criteria are required to support a causal claim regarding the therapeutic benefits of these skills: temporal precedence and non-spuriousness (Feeley, DeRubeis, & Gelfand, 1999). The principle of temporal precedence is frequently violated in psychotherapy research testing process-outcome associations (Webb, DeRubeis, & Barber, 2010). To control for temporal confounds, a given process variable—such as CBT skill use—must be shown to predict subsequent symptom improvement. The majority of prior research testing skill-outcome associations have relied on one or two concurrent assessments (e.g., only pre-treatment and post-treatment) of CBT skills and depressive symptoms (for exceptions, see Hawley et al., 2017; Webb et al., 2016). Within such study designs, a significant association between skills and symptom change could be due to skill use causing symptom improvement or vice-versa (i.e., a failure to establish temporal precedence).

In addition to demonstrating a statistical association and temporal precedence, the third criterion required to support a causal claim is non-spuriousness. A process-outcome association can be considered non-spurious when third variable confounds are ruled out. This last criterion is very challenging to address in observational (non-experimental) studies testing the association between a given psychotherapy process variable and symptom improvement. Specifically, even within studies that observe a significant association between the use of CBT skills and subsequent symptom change, there may be certain stable between-patient characteristics that account for both relatively higher reported use of skills and greater symptom improvement (i.e., a third variable confound). For example, some participants may be more susceptible to study demand characteristics and accordingly, may be more likely to inflate their reports of both skill use and symptom improvement (McCambridge, Bruin, & Witton, 2012). Additionally, certain personality characteristics/disorders have been linked with worse CBT outcomes (Fournier et al., 2008) and could influence reports of CBT skill use. Thus, a significant skill-outcome association could be spurious rather than due to a causal effect of skills on outcome. Unless relevant confounding between-patient characteristics are modeled, a spurious relationship is possible (but see Pearl (2009) regarding blocking “back-door” pathways from the mediator to the outcome).

Recent statistical advances in centering (Howard, 2015) and detrending (Curran & Bauer, 2011) allow for the disentangling of between-patient (i.e., variance between patients in mean levels of skills) and within-patient (i.e., variance within patients in use of skills during treatment) effects of skills on outcome. As illustrated in Figure 1, there may be a significant between-patient effect of skills on outcome, in the absence of a within-patient effect (Panel A) or vice-versa (Panel B). It should be noted that although Panel B illustrates a significant association between within-patient variance in skills and outcome (r = .40, p = .035), conventional analyses (i.e., not disaggregating within-patient and between-patient effects) would conclude that there is no association between skills and outcome (r = .16, p = .410). Importantly, by specifically testing whether within-patient variability in skills predicts outcome, one important source of spuriousness (i.e., differences between patients in stable characteristics) can be ruled out (Zilcha-Mano, 2017). These approaches have not been used to disaggregate CBT skill-outcome associations. In the current study, we tested whether within-patient and/or between-patient variance in skills predicted depressive symptom improvement within a 12-week clinical trial of CBT for depressed adolescents. Importantly, skill use was measured repeatedly over the course of treatment (sessions 1, 3, 6, 9, 12) from both the patient’s and therapist’s perspective allowing to test: (1) therapist-patient agreement in report of skills, and (2) the extent to which CBT skills predicted subsequent symptom change. First, we hypothesized that the association between patient-reports and therapist-reports of skills would be small and non-significant. Second, and consistent with a causal effect of skills on symptom change, we expected that within-patient, but not between-patient, variance in both patient-reported and therapist-reported cognitive and behavioral skills would predict symptom improvement.

Figure 1.

Simulated data for four patients (seven time points per patient) depicting a between-patient (dark line), but no within-patient (dotted lines), effect of skills on symptom improvement (Panel A) vs. a within-patient, but no between-patient, effect of skills on outcome (Panel B). Circles represent skill scores for each patient at each time point, and black squares refer to each patient’s mean skill score.

Method

Participants

Patients.

Participants were 33 female adolescents aged 13–18 years (Mean=15.97; SD = 1.67) with English fluency, who met DSM-IV criteria for a current major depressive episode based on The Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version (K-SADS-PL; Kaufman et al., 1997). Fifty-eight percent of the sample had recurrent MDD, with a median number of prior episodes of 3. Thirteen participants in the MDD group were on SSRIs. As noted below (see Analytic Strategy), medication status was included as a covariate in all models. See Supplementary Methods for additional details.

Therapists.

Nine therapists (6 females and 3 males) delivered 12 weeks of CBT (one 50- minute session per week) (Auerbach, Webb, & Stewart, 2016). Four of the therapists were Ph.D. clinical psychologists, and five were advanced Ph.D. candidates in clinical psychology programs. Therapists received weekly 1-hour individual supervision by licensed clinicians (CAW & RPA), and supervisors reviewed videotaped sessions.

Measures

Beck Depression Inventory (BDI-II; Beck et al., 1996).

The BDI-II is a commonly used 21-item self-report measure assessing depressive symptoms, with excellent psychometric properties (Beck et al., 1996). Participants completed the measure at each session (i.e., weekly) and were instructed to report on their symptoms over the past week. Scores for each item range from 0 to 3, with higher scores indicating higher levels of symptoms. Internal consistencies for the present sample had the following range across assessment timepoint (α = .90 – .94).

Skills of Cognitive Therapy (SoCT; Jarrett et al., 2011).

We utilized the patient-report and therapist-report versions of the SoCT, which consist of 8 items assessing a patient’s understanding and use of both core cognitive skills and behavioral skills. Items are scored on a 5- point scale (1=never to 5=always or when needed) and inquired about skill use during the past week.¹ Previous research suggests that the SoCT can be rated reliably (Jarrett et al., 2013, 2011). Internal consistencies for the present sample had the following range across assessment timepoint for the patient-rated (α = .84 – .89) and therapist-rated (α = .61 – .88) measures.

Procedures

The Partners Healthcare IRB approved this study. Assent was obtained from participants between the ages 13–17, while written consent was obtained from 18-year-olds and their parents. During an initial assessment day, participating adolescents were administered the K-SADS-PL to assess current and past Axis I disorders according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM–IV–TR; American Psychiatric Association, 2000) and completed self-report measures, including the BDI. Following this assessment, participants initiated 12 weekly sessions of CBT based on the following manual (Auerbach, Webb, & Stewart, 2016) (see Supplementary Methods). Participants completed the BDI at the start of each session. Therapists and patients completed the SoCT at the end of sessions 1, 3, 6, 9, and 12. The dataset included 136 observations of patient-rated CBT skills (22 subjects had 4–5 timepoints) and 137 observations of therapist-rated CBT skills (23 cases with 4–5 timepoints).

Analytic Strategy

Given the nested (hierarchical) structure of the data (i.e., sessions nested within individuals), and similar to previous studies (Strunk, Cooper, Ryan, DeRubeis, & Hollon, 2012; Webb et al., 2016), we used SAS (9.2) mixed procedure with maximum likelihood estimation and a heterogeneous autoregressive covariance structure. Specifically, to test the association between predictor variables (i.e., CBT cognitive skills and behavioral skills [SoCT]) and depressive (BDI) symptom change over time, lagged BDI scores for each patient served as the dependent variable (i.e., BDI at Session T+1), with BDI scores at the previous session (Time T) entered as covariates. Cognitive/behavioral skills scores were entered as our predictor variables (Time T). That is, the latter model uses repeated assessments to statistically estimate the relation between predictor measures at a given session (Time T; i.e., sessions 1, 3, 6, 9, and 12) and BDI scores the next session, one week later (Time T+1; i.e., sessions 2, 4, 7, 10, and one week after the final session 12), adjusting for BDI scores at the same session as the predictor variable assessment (Time T). To control for the influence of prior symptom change, a residualized prior change score was included as a covariate (i.e., BDI at Time T, adjusting for BDI at pre-treatment). In addition, the above models control for age, medication status (39.4% currently prescribed antidepressant), BDI scores at the baseline assessment and the linear effect of time. To limit the number of predictor variables, two models were run: one with patient-rated predictor variables and a second with therapist-rated predictors. Next, we implemented statistical procedures to disaggregate within-patient and between-patient effects for both patient- and therapist-reports of skills (Curran & Bauer, 2011; Wang & Maxwell, 2015; Supplemental Methods). Importantly, this approach yields independent coefficients for within- and between-patient effects, which are included simultaneously in the same model.²

Results

As shown in Table 1 (bottom panel), patient-reported use of behavioral—but not cognitive—skills predicted depressive symptom improvement (b = −2.05; t(84) = −2.87; p =.005). Even when adding the two therapist-rated variables (cognitive and behavioral skills) to the above model, patient-rated behavioral skills remained a significant predictor of symptom change (b = −2.03; t(76)= −2.75; p = .007).

Table 1.

The relations between skills and depressive symptom improvement

Predictor	Rater	Parameter Estimate (b)	Standard Error (SE)	t-value	p-value
CBT-C		−0.09	.81	−0.11	.911
	Therapist
CBT-B		0.26	.70	0.37	.712
CBT-C		0.71	.86	0.82	.413
	Patient
CBT-B		−2.05	.71	−2.87	.005

Note: CBT-C = Skills of Cognitive Therapy (SoCT) scale: cognitive items; CBT-B = SoCT scale: behavioral items. All models control for pre-treatment depression severity, symptoms at the time at which the predictor was assessed, prior symptom change, age, medication status and time (linear term). Degrees of freedom (df) for patient model (df=84) and therapist model (df=82). In the above table, negative t values indicate that higher scores on the predictor variable are related to relatively larger improvements in depressive symptoms.

Disaggregating within-patient vs. between-patient effects.

First, intraclass correlation coefficients (ICCs) were computed to estimate the proportion of variance in skills accounted for by patients (i.e., between-patient variance) vs. time/session (i.e., within-patient variance). ICCs for patient-rated behavioral skills and cognitive skills were .34 and .39, respectively, indicating that 34–39% of the variance in skills was between-patient variation. Thus, the majority of the variance in skills scores (61%−66%) was within patients (Figure 2a). For therapist-rated cognitive and behavioral skills, the corresponding ICCs were .10 and .13, indicating that from the therapists’ perspective, variance in skills was largely within patients. Table 2 presents correlations among between-patient and within-patient predictor variables. Importantly, when disaggregating within-patient and between-patient effects, within-patient variability in both patient-rated (b =−2.55; t(84)=−2.28; p =.025) and therapist-rated (b =−2.41; t(82)=−2.17; p =.033) behavioral skills predicted symptom change (Table 3). Specifically, during weeks when patients exhibited greater behavioral skills than their average levels, their depressive symptoms were relatively lower the following session (see Supplement for analyses examining the effect of removing covariates, testing associations with prior symptom change, including the therapeutic alliance as a predictor, and with missing values imputed via a Random Forest procedure).

Figure 2.

Behavioral (blue) and cognitive (green) scores at five timepoints for patient-report (Panel A) and therapist-report (Panel B) Skills of Cognitive Therapy (SoCT) measure. For both panels, thick blue and green lines represent mean values.

Table 2.

Correlations among between-patient (below diagonal) and within-patient (above diagonal) predictor variables

Variable	1	2	3	4
1. CBT-C (p)	-	.53(5/5)	.20(0/5)	.03(0/5)
2. CBT-B (p)	.77**	-	.05(0/5)	.20(1/5)
3. CBT-C (t)	.22	.35*	-	.33(2/5)
4. CBT-B (t)	.27	.38*	.76**	-

Note: Correlations for between-patient scores are presented below the diagonal. Above the diagonal, values are the average correlations of within-patient predictor variables across the five sessions. Fractions in superscript indicate at how many of the sessions was the correlation significant. Given that the sampling distribution of rs is known to be skewed, calculations were carried out on Fisher’s Z values (Rosenthal, 1991) and then converted back to rs.

^*p < .05;

^**p < .001

Table 3.

Disaggregating within-patient and between-patient effects of skills

Predictor	Rater	Parameter Estimate (b)	Standard Error (SE)	t-value	p-value
CBT-C
Within		1.90	1.26	1.51	.134
Between		−3.59	3.47	−1.03	.310
	Therapist
CBT-B
Within		−2.41	1.11	−2.17	.033
Between		−3.61	3.18	−1.13	.267
CBT-C
Within		−0.42	1.30	−0.33	.743
Between		−3.15	2.94	−1.07	.293
	Patient
CBT-B
Within		−2.55	1.12	−2.28	.025
Between		−1.56	2.59	−0.60	.551

Note: CBT-C = Skills of Cognitive Therapy (SoCT) scale: cognitive items; CBT-B = SoCT scale: behavioral items All models control for pre-treatment depression severity, symptoms at the time at which the predictor was assessed, prior symptom change, age, medication status and time. In the above table, negative t values indicate that higher scores on the predictor variable are related to relatively larger improvements in symptom scores.

Discussion

To our knowledge, this is the first study to disaggregate within-patient vs. between-patient effects of skills on symptom change in either adults or youth. As noted, examining within-patient variance in skills, while accounting for between-patient variability, controls for one potentially important third variable confound (i.e., stable between-patient characteristics that influence both skills use and symptom change). Causal claims about the role of CBT skills in contributing to symptom change implicitly assume a within-patient effect. Thus, it is critical to disentangle and model these sources of variance to test a causal claim (Curran & Bauer, 2011; Zilcha-Mano, 2017). Second, in contrast to the bulk of prior process-outcome studies (Webb et al., 2010), we controlled for temporal confounds by predicting subsequent symptom change. Third, we simultaneously considered both the therapist’s and patient’s perspective on skills. Finally, most prior studies testing the association between a given psychotherapy process variable (e.g., patient skills, therapist treatment adherence) and symptom change focus on relatively long-term predictions. Namely, process variables are typically assessed in the midst of treatment (e.g., session/week 3) and are correlated with scores on a post-treatment symptom measure assessed several weeks or months later (e.g., session/week 12), while statistically controlling for baseline values. Even if a causal relationship exists, it may be obscured due to the reliance on such a relatively long time-span between assessments of the predictor and outcome, and the host of other potential influences on symptoms that occur during the intervening period. We tested whether CBT skills predicted session-to-session symptom change.

As presented in Table 2, therapist and patient ratings of CBT skills (rs = .20 −.38) showed small to moderate associations. Consistent with a causal claim that the acquisition and use of behavioral skills contributes to depressive symptom change, both therapist- and patient-reported data revealed that within-patient variance in behavioral—but not cognitive—skills predicted session-to-session symptom change. In other words, even when controlling for average between-patient differences in skill use, the extent to which patients are using relatively greater levels of behavioral (but not cognitive) skills than their average predicts greater symptom improvement from the current to the next session. There are several interpretations of these findings. First, cognitive strategies are more challenging to learn and effectively deploy relative to simpler behavioral techniques. However, both patient-report (Figure 2a) and therapist-report (Figure 2b) data indicate that patients did increase their use of cognitive skills over the course of treatment. Despite the apparent acquisition and use of both cognitive and behavioral skills, only the latter predicted depressive symptom change. It is important to highlight that the mean intake depression score for this adolescent sample was in the severe range (mean intake BDI-II = 31.7). There is a growing body of clinical trial (Dimidjian et al., 2006) and process-outcome (Webb et al., 2016) research indicating greater therapeutic benefits of behavioral activation skills for those with higher–but not lower–levels of depressive symptoms, which may help account for our findings being specific to behavioral skills. In addition, from a neurodevelopmental perspective, cognitive work within CBT requires several higher-order cognitive abilities (e.g., metacognition, abstract and flexible thinking, cognitive control), yet the necessary prefrontal circuitry subserving these abilities does not reach full maturity until the twenties (Giedd, 2004). Thus, behavioral skills may be better developmental fit, at least for many adolescents. Finally, cognitive skills may predict a lowered risk of relapse, which was not tested in this study.

Several limitations should be noted. First, the study was exclusively focused on adolescent females ages 13–18. Thus, it is unclear whether the observed pattern of findings would generalize to depressed males or to other age ranges. Second, although our analytic approach controlled for temporal confounds and disaggregated within-patient from between-patient variance in skills, the study relied on a non-experimental (observational) design, limiting causal conclusions. Third, despite the use of repeated assessments, sample size was relatively small (n=33), in particular for the between-subject effects. Fourth, measures of predictors and outcome include a certain amount of error, which may significantly attenuate estimates of predictor-outcome associations. Fifth, ratings of therapist adherence were not collected. Finally, and similar to the vast majority of process-outcome research, skills were assessed via self-report. We are aware of no study in adolescents that has tested the association between skills and symptom change using observational coding of skills. There are, however, a few studies in adults using trained raters to code CBT skills from videotaped sessions (e.g., the Performance of Cognitive Therapy Skills measure) (Strunk et al., 2014; Webb, DeRubeis, et al., 2012). These limitations notwithstanding, the present findings suggest that focusing on the acquisition and use of behavioral CBT skills may be particularly therapeutically beneficial for depressed adolescent females. In addition, the integration of brief patient-report and/or therapist-report measures of CBT skills in clinical practice may be useful to gauge the extent to which depressed teens are integrating these skills in their day-to-day lives.

Figure 3.

Scatterplot of the association between therapist-report and patient-report cognitive skills (Panel A) and behavioral skills (Panel B). For both panels, black line represents overall correlation, whereas colored lines represent the correlation within each patient.

Public Health Significance Statement.

A growing body of research supports the efficacy of cognitive behavioral therapy (CBT) for depressed adolescents. However, no study has tested the extent to which depressed adolescents acquire cognitive and behavioral skills in CBT and, critically, whether these skills predict symptom improvement. The present study suggests that the acquisition and use of behavioral – but not cognitive – skills predicts symptom change.