Assessing Patients’ Cognitive Therapy Skills: Initial Evaluation of the Competencies of Cognitive Therapy Scale

Abstract

In Cognitive Therapy (CT), therapists work to help patients develop skills to cope with negative affect. Most current methods of assessing patients’ skills are cumbersome and impractical for clinical use. To address this issue, we developed and conducted an initial psychometric evaluation of self and therapist reported versions of a new measure of CT skills: the Competencies of Cognitive Therapy Scale (CCTS). We evaluated the CCTS at intake and post-treatment in a sample of 67 patients participating in CT. The CCTS correlated with a preexisting measure of CT skills (the Ways of Responding Questionnaire) and was also related to concurrent depressive symptoms. Across CT, self-reported improvements in CT competencies were associated with greater changes in depressive symptoms. These findings offer initial evidence for the validity of the CCTS. We discuss the CCTS in comparison with other measures of CT skills and suggest future research directions.

Cognitive Therapy (CT) is an efficacious treatment for depression, with acute efficacy that is comparable to antidepressant medication (DeRubeis et al., 2005; Strunk & DeRubeis, 2001). Nonetheless, the mechanisms whereby CT achieves its effects are not yet well understood. One potential mechanism involves the acquisition of coping skills over a course of CT (Beck, 1967, 1976). In CT, therapists teach patients cognitive and behavioral skills that they can use during treatment and continue to use after treatment ends (Beck, Rush, Shaw, & Emery, 1979). Previous research has shown that patients’ acquisition of CT skills was related to concurrent depressive symptom reductions (Barber & DeRubeis, 2001; Connolly Gibbons et al., 2009). Moreover, following a successful course of CT, patients’ CT skills predicted lower risk of relapse across a one year follow up period (Strunk, DeRubeis, Chiu, & Alvarez, 2007).

Cognitive Therapy Skills

Cognitive therapists work to foster patients’ coping skills so that patients are prepared to use these skills to cope more effectively with negative moods (Beck et al., 1979). CT patients learn to identify their automatic thoughts, recognize the connection between their thoughts and moods, and then assess the accuracy of their thoughts. As Hundt, Mignogna, Underhill, and Cully (2013) state in their review of CT skills, learning to re-evaluate the accuracy of one’s automatic thoughts is arguably the most central CT skill. As part of evaluating the accuracy of thoughts, patients may examine available evidence, generate potential alternative explanations for events, and conduct behavioral experiments. Both cognitive and behavioral skills are essential parts of CT. A key focus of behavioral skills in CT is to increase patients’ sense of pleasure or accomplishment. Patients learn to engage proactively in pleasurable activities to boost their mood and to break down problems into smaller parts. These behavioral strategies may be used to promote symptom relief directly or as an effort to provide patients with information to help them re-evaluate maladaptive thoughts and beliefs. Once substantial therapeutic progress has been made, therapists work with patients to modify more directly their core beliefs and schemas.

Several studies have examined the empirical relation of CT skills and therapeutic outcomes using existing CT skills measures (for a review of skill measures, see Hundt et al., 2013). Multiple measures of CT skills have been proposed recently, including two measures evaluated in large samples: the Skills of Cognitive Therapy (SoCT; Jarrett, Vittengl, Clark, & Thase, 2011) and the Cognitive-Behavioral Therapy Skills Questionnaire (Jacob, Christopher, & Neuhaus, 2011).¹ Each of these scales has been shown to predict improvement in depressive symptoms during CT.

Because we thought one of the primary applications of a CT skills measure would be the prediction of relapse or recurrence, we focused our attention on the Ways of Responding questionnaire (WOR) and the Performance of Cognitive Therapy Skills scale (PCTS). Each of these measures has been shown to predict relapse following a successful course of CT (Strunk et al., 2007). The validity of other recently developed self-report CT skill measures has yet to be evaluated with regard to their relation to the WOR, PCTS, or outcome following successful treatment. Since Barber and DeRubeis (1992) introduced the WOR, it has been used to examine CT skills in a number of contexts (Kapson, Leddy, & Haaga, 2012; Webb et al., 2012; Connolly Gibbons et al., 2009). Scoring the WOR requires trained coders to rate the level of CT skill evident in patients’ responses to the series of hypothetical events included in the WOR. Strunk and colleagues developed the PCTS for rating patients’ mastery and independent use of CT skills among treatment responders on the basis of observations of CT sessions. While methodologically distinct, these two measures both involve lengthy rating procedures that make them labor intensive for research and impractical for clinical use. Based partly on training times reported in Strunk et al., we estimate that the WOR requires three raters to complete 14 hours of training each and spend a total of 45 minutes rating each protocol. The PCTS requires two raters to complete 70 hours of training each and spend a total of 9.3 hours rating each patient. A brief and readily scored measure of these skills would have substantially greater practical utility. With these considerations in mind, we sought to develop and evaluate such a measure.

Current Methods of Assessing CT Skills

Before developing a new assessment method, it is important to understand established methods of assessing CT skills. We focus on the WOR and the PCTS, as both of these measures have been empirically related to lower risk for relapse. The WOR yields both a total and an overall quality score. WOR scores are generated by trained raters who evaluate participants’ responses to a series of hypothetical scenarios. Trained coders parse participants’ open-ended responses into individual thought units and indicate whether each parsing reflects a positive (e.g., generating alternative explanations) or negative (e.g., placing blame on self) response. The resulting ratings are summarized through a total score, calculated as the number of positive category ratings minus the number of negative category ratings. In addition, raters also provide an assessment of the overall response quality, reflecting the participant’s degree of mastery of CT skills.

As noted previously, the PCTS is a more recently developed observer-rated measure of patients’ performance of these skills both in and between CT sessions (Strunk et al., 2007). The PCTS involves trained raters reviewing therapy sessions and providing ratings of patients’ skills based on their in-session demonstration of these skills and their reported use of CT skills between sessions. Like the WOR, the PCTS also predicted risk of relapse in the year following treatment with CT (Strunk et al.). When entered in the same model, both the WOR and the PCTS were independent predictors of risk for relapse.

Although available evidence suggests the WOR and the PCTS are useful predictors of concurrent reductions in depressive symptoms and lower risk of relapse, these measures both involve a time consuming rating process that makes them challenging for use in research and impractical for clinical use. A valid self-report measure of patients’ mastery and use of CT skills could have significant advantages. In fact, Jarrett and colleagues (2011) made this argument in the paper in which they reported on their effort to develop a new measure of CT skills (the SoCT). Because we were unaware of their work until our own efforts were near completion, we did not collect data on their measure. However, we will return to consider the similarities and differences in our effort and that of Jarrett and colleagues in the discussion section.

Given the labor required for utilizing measures such as the WOR and PCTS, we developed a patient reported 30-item measure: the Competencies of Cognitive Therapy Scale – Self-Report version (CCTS-SR). We also developed a 9-item therapist rated measure: the Competencies of Cognitive Therapy Scale – Therapist Report version (CCTS-TR). The language of the CCTS-SR was chosen carefully so that the measure could be used both pre- and post-treatment. Items were written so that they could be understood by those without any exposure to CT. For both measures, we constructed items to capture CT skills relevant to the three primary components of CT (i.e., behavioral strategies, re-evaluating automatic thoughts, and schema/core belief strategies).

To assess the validity of the self-report and therapist-reported versions of the CCTS, we had five specific objectives. First, we tested the factor structure of the two versions of the CCTS (i.e., self and therapist reported). Second, to evaluate whether the CCTS could detect patients’ acquisition of CT skills over the course of treatment, we examined change on the CCTS-SR from pre- to post-treatment. Third, to determine if the CCTS could differentiate the skills of depressed patients from healthy, never depressed controls, we compared CT skills in controls and depressed patients before and after CT. Fourth, we evaluated the concurrent validity of the CCTS in two ways: (1) examining the relationship between ratings of CT skills on the WOR and the CCTS at corresponding time points; and (2) examining the relation of skill change over the course of treatment as measured by the CCTS and WOR. Finally, we tested the relation of CCTS assessed change in CT skills and symptom reduction over the course of CT.

We also report on secondary analyses among treatment responders only, examining the relation of the CCTS and established CT skills measures. Because the assessment of CT skills may be particularly useful among patients who have responded to treatment (e.g., as predictors of relapse), we examined the concurrent validity of the CCTS in this more limited sample. Specifically, we tested the relation of the CCTS with the PCTS and WOR.

Method

Participants

Two groups participated in this study: a sample of 67 depressed patients who participated in 16 weeks of CT and a sample of 44 never depressed control participants. Patients in the depressed sample met the following inclusion criteria: a diagnosis of current MDD using the Structured Clinical Interview for DSM-IV (SCID; First, Spitzer, Miriam, & Williams, 2002), and agreeing not to make changes to their psychiatric medication regimen over the course of the study. Exclusion criteria were: being under the age of 18 or having a history of Bipolar I disorder or psychosis. To participate as a control, participants were included if they had no history of anxiety disorder, mood disorder or psychosis.

Because one patient never completed a CCTS measure, our sample for analyses was comprised of 66 patients. These patients were predominately female (55%) and Caucasian (85%). The mean age of the depressed group was 36 years (SD = 13.3). A total of 34.9% were on antidepressant medication. The control group was comprised of participants matched to the 44 participants in the depressed sample who completed treatment. Participants were matched on sex, age (within 2.5 years; M = 38, SD = 14.2), and education. For additional description of these samples, see Adler, Strunk, and Fazio (2014).

Measures

Diagnosis

The Structured Clinical Interview for DSM-IV (SCID; First et al., 2002) was used to assess whether patients met formal criteria for current MDD and any other Axis I disorders. Assessment of the reliability of the Major Depressive Disorder designation yielded a kappa coefficient of 1.00 (n = 12).

Depressive symptoms

We collected two measures of depressive symptom severity: the clinician-rated 17-item Hamilton Rating Scale for Depression (HRSD; Hamilton, 1960; Williams 1988), modified to assess atypical symptoms and the 21-item, self-report Beck Depression Inventory- 2nd Edition (BDI-II; Beck, Steer, & Brown, 1996). Higher scores on both measures indicate greater depressive symptoms. Using independent ratings of 30 video recordings of HRSD evaluations, we calculated the intraclass correlation coefficient for HRSD scores to be .99. Internal consistency for both scales was acceptable (for the HRSD, Cronbach’s α was .60 at intake and .77 at post-treatment; for the BDI, α was .86 at intake and .91 at post-treatment).

Cognitive Therapy skills

The WOR (Barber & DeRubeis, 1992) involves assessing CT skills through ratings of patients’ responses to a series of challenging hypothetical scenarios. Respondents are instructed to record any subsequent thoughts or behavioral reactions they would have about each situation. Using guidelines outlined in the WOR Rater’s Guide (Barber & DeRubeis), three coders rate responses. Responses are parsed into individual thought units, and two independent raters classify each parsing as either a positive or negative response. A third rater resolves any disagreement between the first two raters. A total score is calculated from these ratings by subtracting the number of negative responses from the number of positive responses. An additional summary score, quality, represents the average of the three raters’ individual assessments of the extent that the response would improve an individual’s mood. The intraclass correlation coefficient (ICC) for WOR quality scores (corrected for three raters) was excellent (.97).

Competencies of Cognitive Therapy Scale (CCTS)

We developed two versions of this measure: a 30-item patient version (CCTS-SR; see Appendix A) and a 9-item therapist version (CCTS-TR; see Appendix B). As we later report, we dropped a single item from the CCTS-SR due to low factor loadings, resulting in a 29-item measure. Both measures are designed to assess the patients’ mastery of the skills that therapists help patients develop in CT. Respondents rate each item on the CCTS-SR on a 1 (not at all) to 7 (completely) scale with respect to their degree of skill use in the past two weeks. The CCTS-TR uses therapist ratings to evaluate a patient’s ability, independence, and frequency of use of behavioral activation, automatic thoughts, and core belief related CT strategies on a scale from 0 (none) to 6 (extensive).

Performance of CT Strategies (PCTS)

The PCTS (Strunk et al., 2007) is an observer rated measure of patients’ skills, understanding, and use of strategies taught in CT. The measure includes items relevant to behavioral activation, automatic thought evaluation, and schema change. Raters are to evaluate skill on the basis of patients’ in session demonstration and reported between session use of CT skills. Each item is rated on a Likert-type scale ranging from 0 to 6. Ratings of 4 or higher indicate independent use of the skill assessed by that item. All items are summed to yield a total score reflecting patients’ overall use of CT skill. Similar to the method of Strunk et al. (2007), two graduate level raters (blind to outcome) rated the second session (to serve as a baseline skill score), and then provided a rating after reviewing sessions from week 12, 14, and the last session before the post-treatment evaluation (in chronological order). PCTS ratings were completed for patients who completed and responded to CT treatment. Response was defined as a HRSD score of less than 12 and not meeting current MDD criteria. The ICC for PCTS scores (corrected for two raters) was .82. Cronbach’s alpha was .93.

Dysfunctional attitudes

The Dysfunctional Attitudes Scale (DAS; Weissman, 1979) is a 40-item self-report instrument measuring maladaptive beliefs (“I cannot be happy unless most people I know admire me”) common among depressed patients as proposed in Aaron Beck’s theory (Beck, 1967, 1976). The DAS has shown adequate test retest reliability over 2 and 3 month follow-up periods (rs of .84 and .74, respectively; Weissman). Depressed respondents have been found to score as having more dysfunctional attitudes than non-depressed respondents (Gotlib, 1984). While the DAS does not directly assess effortful CT skill usage, dysfunctional attitudes as assessed by the DAS have been related to lower CT skills (see Adler et al., 2014). In this study, Cronbach’s alphas for the DAS were .94 and .93 at intake and post-treatment evaluations, respectively.

Procedures

CT was provided in accord with the procedures described by Beck and colleagues (1979). Consistent with other clinical trials (e.g., Elkin et al., 1989), the duration of treatment was 16 weeks. Therapists were four advanced graduate students (three women and one man). Sessions were provided once per week, with patients who initially scored a 20 or greater on the HRSD being offered sessions twice weekly for the first four weeks. At the intake evaluation, we administered measures of depressive symptoms (BDI-II, HRSD), CT skills (WOR, CCTS-SR), and dysfunctional attitudes (DAS). At the post-treatment evaluation (16 weeks after the intake evaluations), we administered the BDI-II, HRSD, WOR, CCTS-SR, CCTS-TR, and DAS. Following each intake evaluation (when therapists were randomly assigned to cases), evaluations were completed by a clinician other than the patients’ therapist.

CCTS-SR scores were completed by 66 of the 67 depressed patients who met eligibility criteria and enrolled in the study. A total of 44 patients completed treatment. The 44 control participants were matched to these 44 treatment completers using the criteria previously described. Among the 66 patients with some CCTS data, we were missing CCTS-SR scores for a single patient at intake as well as for another patient at post-treatment, bringing our final sample size for CCTS-SR scores at intake to 65 and at post to 43. CCTS-TR scores were available for 42 patients who completed treatment.

Results

CCTS Factor Structure and Relation Between Self-Report and Therapist Versions

We used Exploratory Factor Analysis (EFA) to examine the factor structure of the CCTS-SR at the intake and post-treatment assessments and the CCTS-TR at post-treatment. For each EFA, we conducted a principal factor analysis. We determined the number of factors to retain based on an examination of the scree plot and parallel analysis. In each case, both the scree plot and parallel analysis suggested one factor solutions.² In analyses of the CCTS-SR, a single item demonstrated a low factor loading at intake (.14) and was therefore removed from the item pool. Items and factor loadings for the CCTS-SR and CCTS-TR are provided in Appendix A and B, respectively. At the intake and post-treatment assessments, Cronbach’s alphas for the 29-item CCTS-SR were .93 and .97, respectively. Item-total correlations ranged from .36 to .77 at intake and .56 to .85 at post-treatment. Cronbach’s alpha for the CCTS-TR was .96, with item-total correlations ranging from .78 to .91.

We then examined the correlation between self-report and therapist-reported versions of the CCTS. At the post-treatment assessment (where both CCTS-SR and CCTS-TR scores were available), the two CCTS versions were significantly correlated (r = .54, n = 40, p = .0003).

CT Competencies: Changes over the Course of CT and Comparisons with a Control Sample

At the intake evaluation, the mean CCTS-SR was 93.97 (SD = 26.36, n = 65). The mean score at post was 132.22 (SD = 31.60, n = 42). For the CCT-TR (rated by therapists at post treatment), the mean score was 25.21 (SD = 12.70, n = 42). To test whether patients reported significant improvements in CT competencies, we used a paired t-test comparing CCTS-SR scores at intake and post-treatment. As shown in Figure 1, patients reported a large, significant increase in CT Skills (d = 1.41, t (41) = 6.10, p < .0001). Also shown in the figure, we compared CCTS-SR scores across the depressed patient and control comparison samples. For these comparisons, we examined patients who completed treatment and the matched control sample. At the intake assessment, patients reported substantially and significantly fewer CT skills than controls (d = 1.29, t(84) = 5.99, p < .0001).³ We then compared the control sample to CCTS-SR scores of the depressed sample at post-treatment. In this comparison, the difference between patients and controls was small and not significant (d = .07, t (84) = −.32, p = .75). Thus, patients reported lower CT skills than never-depressed controls prior to CT, but at the conclusion of treatment, patients exhibited scores comparable to those reported by healthy control participants.

Figure 1.

CCTS-SR Scores among CT Patients and Healthy Controls

Note. CCTS-SR = Cognitive Therapy Competencies Scale-Self Report; CCTS-TR = Cognitive Therapy Competencies Scale-Therapist-Report.

Sample sizes for each comparison range from 42–43 due to one patient with a missing CCTS-SR score at intake and another patient missing a score at post. Letters above bars indicate statistical significance of mean differences. Letters that differ indicate means differ at the p < .05 level. Error bars reflect standard error of the mean estimate.

Correlation of the CCTS with CT Skills, Dysfunctional Attitudes, and Depressive Symptoms

To examine the nomological network of CCTS assessed CT skills, we evaluated the concurrent associations of CCTS scores with other measures of CT skills and dysfunctional attitudes at intake and post-treatment evaluations. Because the CCTS-TR was administered only at post-treatment, correlations involving the CCTS-TR were necessarily limited to this assessment. As Table 1 displays, we found a significant, moderate correlation between the CCTS-SR and another CT skills measure, the WOR. This correlation provides evidence for the concurrent validity of the CCTS-SR.

Table 1. Correlations of pre- and post-treatment CCTS assessed CT skills with other measures of CT skills and depressive symptoms.

	Intake		Post
	CCTS-SR		CCTS-SR		CCTS-TR
	r	n	r	n	r	n
Measures of Cognitive Therapy Skills
WOR-Quality	.30*	63	.37*	43	.35*	42
Dysfunctional Attitudes
DAS	−.17	64	−.30†	43	−.27†	40
Depressive Symptoms
BDI-II	−.39**	63	−.49**	43	−.59**	40
HRSD	−.22†	65	−.46**	43	−.41**	41

Note. BDI-II = Beck Depression Inventory- 2nd Edition; CCTS-SR = Cognitive Therapy Competencies Scale-Self Report; CCTS-TR = Cognitive Therapy Competencies Scale-Therapist-Report; DAS = Dysfunctional Attitudes Scale; HRSD = Hamilton Rating Scale for Depression; WOR = Ways of Responding.

The CCTS-TR was administered only at the post-treatment assessment.

^†p < .10;

^*p < .05 ;

^**p < .01

Dysfunctional attitudes were not related to CCTS-SR scores at intake; at post-treatment, both CCTS-SR and CCTS-TR scores were related to dysfunctional attitudes at the level of a non-significant trend (see Table 1). Thus, while there was some evidence of a possible association of the CCTS-SR and CCTS-TR with dysfunctional attitudes at post-treatment, CCTS-SR and CCTS-TR were generally distinct from depressive cognition as assessed by the DAS.

Next, we examined the association of the CCTS and depressive symptoms. At intake, the CCTS-SR was significantly related to lower depressive symptoms on the BDI and was related to lower HRSD scores at the level of a non-significant trend (see Table 1). At the post-treatment assessment, both the CCTS-SR and CCTS-TR were significantly associated with the BDI and HRSD. Thus, there was some evidence of a relation of CT skills and depressive symptoms at the intake assessment. At the post-treatment assessment, CT skills were consistently related to depressive symptoms.⁴

Given the potential importance of measuring CT skills among treatment responders, we conducted secondary analyses of post-treatment scores among the 29 patients who met response criteria. As was true in the larger sample, in this subset of patients, the WOR quality scores demonstrated a moderate relation with the CCTS-SR (r = .47, n = 28, p =.01) as well as with the CCTS-TR (r = .45, n = 27, p = .02). In addition, PCTS ratings were significantly related to both the CCTS-SR (r = .53, n = 28, p =.004) and the CCTS-TR (r = .75, n = 27, p < .0001). Thus, even when limited to treatment responders, the CCTS demonstrated convergent validity with other measures of CT skills.

Relation of the CCTS and Change in Depressive Symptoms

Next, we examined the relation of pre- to post-treatment changes in CT skills (as assessed by the WOR and CCTS-SR) and pre- to post-treatment changes in depressive symptoms (as assessed by the HRSD and BDI). For these analyses, we created residualized change scores for each of the CT skill measures by utilizing the residuals from regression models in which a post-treatment skill measure served as the dependent variable and pre-treatment scores on that same measure was entered as a predictor. We examined each of these CT skill residualized change scores as predictors of pre- to post-treatment changes in depressive symptoms. In these models, we entered a post-treatment depressive symptom measure as the dependent variable and entered that same depressive symptom measure at pre-treatment as a covariate. Using this approach, there were non-significant trends for residualized change in WOR quality to predict depressive symptoms as measured by both the HRSD and the BDI (for the HRSD: β = −.31, t(36) = −1.91, p = .07; for the BDI: β = −.30, t(36) = −1.81, p = .08). Residualized change on the CCTS-SR was significantly related to changes in depressive symptoms for both the HRSD and the BDI (for the HRSD: β = −.49, t(36) = −3.27, p = .003; for the BDI: β = −.49, t(36) = −3.24, p = .003).

We then examined the CCTS-SR as a predictor of symptom change beyond that accounted for the WOR or the PCTS. First, we examined the CCTS-SR in models including the WOR. We utilized a regression model in which we examined residualized (intake to post) change in WOR quality and residualized (intake to post) change in CCTS-SR scores as predictors of depressive symptoms at post, while controlling for depressive symptom scores at intake. In this model predicting HRSD symptom change, the CCTS-SR remained a significant predictor of reductions in depressive symptoms (β = −.38, t(39) = −2.40, p = .02) whereas the WOR was no longer a significant predictor (β = −.16, t(39) = −.99, p = .33). Similarly, in a model predicting BDI change, the CCTS-SR emerged as a significant predictor (β = −.40, t(39) = −2.54, p = .02) and the WOR was not significantly predictive of symptom change (β = −.14, t(39) = −.92, p = .36). Finally, we examined the CCTS-SR in models including the PCTS as an additional predictor (necessarily limited to the sample of treatment responders). In these models, we entered the CCTS-SR and PCTS as predictors of post depressive symptoms controlling for intake depressive symptoms. In the model for the BDI and HRSD, neither predictor remained significant (all ps > .2).

Does the CCTS merely measure symptom severity?

Although CCTS items were designed to assess CT skills, one might be concerned that the CCTS-SR could merely reflect symptom severity. In light of this concern, we examined whether CTCS scores were related to the WOR after controlling for depressive symptoms (i.e., HRSD and BDI scores). To maximize power (and avoid problems with a restricted range of depressive symptom severity for such an analysis), we examined the full sample of patients as well as the control sample. Because variables were not normally distributed when examined in this combined sample, we used Spearman correlations. Specifically, we examined the relationship between the CCTS-SR and the WOR after partialling out either BDI or HRSD scores. The CCTS-SR remained significantly correlated with WOR quality scores (r_s = .22, p = .02, n = 105) after partialling out the BDI. The CCTS-SR and WOR quality scores at intake were also significantly related after partialling out HRSD scores (r_s = .30, p = .002, n = 107).⁵ These results suggest that the CCTS-SR accounts for variance in skills as measured by the WOR even after accounting for variance related to depressive symptom severity.

Discussion

Our initial evaluation of the CCTS-SR and CCTS-TR suggests these measures show promise as brief, easy-to-score measures of CT skills. For both the CCTS-SR and the CCTS-TR, factor analyses suggested one factor solutions. Using the CCTS-SR, we found patient rated CT skills improved significantly and markedly over the course of CT. Moreover, while patients at intake were significantly less skilled than never depressed controls, after 16 weeks of treatment, patients reported CT skills no longer differing from those reported by control participants. In addition, two sets of findings attest to the construct validity of the CCTS. First, CCTS scores were consistently correlated with another independent measure of CT skills (i.e., the WOR). In fact, in a combined sample of patients and controls, this relationship remained significant even after controlling for depressive symptoms. Second, the CCTS was significantly related to depressive symptoms as assessed by the HRSD and BDI in five of the six tests we examined across time points. We also found that patients who reported greater gains in CT skills experienced greater reductions in depressive symptoms. In fact, in our analyses, the relation of change in CT skills and change in depressive symptoms was only significant when assessing skills with the CCTS-SR; this relation was a non-significant trend when using the WOR. Thus, taken together, this evidence suggests the newly developed CCTS assessments of CT skills show promise as brief, easy-to-use measures of CT skills.

The large positive changes in CT skills patients reported across the course of CT using the CCTS-SR (d = 1.41) are consistent with the magnitude of change in CT skills estimated using other measures. Specifically, we report elsewhere that the CT patients in the present sample exhibited large gains in CT skills as assessed by the WOR (d = 1.53; Adler et al., 2014). These changes are also consistent with the large changes in WOR scores Barber and DeRubeis (2001) identified among CT patients in their initial research using the WOR. Insofar as the results for the CCTS correspond to the results we obtained using another assessment method, the results bolster our confidence in the CCTS as a measure of CT skills.

In addition, we report elsewhere that patients’ intake WOR scores were markedly lower than those in the healthy control sample (d = 1.82; Adler et al., 2014). Like the differences found with the WOR, the CCTS-SR showed a large gap between the skills of patients before treatment and the greater skills reported by healthy controls (d = 1.29). Other than this work, little research has been conducted to examine whether the skills taught in CT are comparable to strategies that healthy non-depressed people use to cope. In a recent study involving a non-clinical sample, Adler, Conklin & Strunk (2013) found participants who reported greater use of CT skills subsequently showed less depressive symptom reactivity to a series of repeated stressors. In combination with the finding that patients at post-treatment reported CCTS rated skills comparable to healthy control participants, emerging evidence now suggests that the skills patients acquire over the course of CT treatment may be comparable to those employed naturally by healthy individuals.

Because we examined the CCTS-TR at only the post-treatment assessment, our evaluation of this version of the CCTS was more limited. Nonetheless, we found support for the usefulness of the CCTS-TR, with this measure being moderately correlated with both the WOR and concurrent depressive symptoms at the post-treatment assessment. As the CCTS-TR is not self-reported, this version of the CCTS-TR may be less likely to be contaminated by any patient biases in reporting. Perhaps the greatest advantage of the CCTS-TR is that it is quite short and easy to use.

As we noted earlier, the WOR and PCTS have been shown to predict risk for relapse following a successful course of CT (Strunk et al., 2007). Unfortunately, although we endeavored to collect follow-up data, attrition over the follow-up period prevented us from being able to provide an adequately powered test of the CCTS-SR and CCTS-TR as predictors of patients’ risk of relapse. Given the potential importance of assessing CT skills among patients who have responded to treatment specifically, we did conduct secondary analyses examining the relation of the CCTS with the WOR and the PCTS among treatment responders. We found moderate correlations between the CCTS and both preexisting skills measures. Future research will need to evaluate the CCTS as a potential predictor of risk of relapse. If short, easy to score measures such as the CCTS do predict relapse, such measures could ultimately prove useful in identifying CT responders who may still be at high risk of relapse (who might benefit from additional intervention).

Across versions and assessments times, the CCTS was largely related to measures of depressive symptoms. In cross-sectional tests, the CCTS was significantly correlated with depressive symptom measures in five of the six tests we examined. Specifically, the CCTS-SR showed small to moderate correlations with depressive symptoms at intake (with the relation of CCTS-SR and HRSD being a non-significant trend) and both the CCTS-SR and CCTS-TR demonstrated significant, moderate correlations with depressive symptoms at post-treatment. In addition, across the course of CT, residualized change in CCTS-SR was significantly correlated with residualized change in depressive symptoms. In the two tests of this relationship, the relation of change in CT skills and depressive symptoms were of moderate size (βs = −.49).

In their review of the literature, Hundt and colleagues (2013) suggested that the validity of some measures of CT skills may be compromised by using items that assess the absence of depressive symptoms (in addition to CT skills). The items of CCTS were written in an effort to avoid the potential overlap of skill assessment with depressive symptoms. Nonetheless, one might be concerned that even in the absence of item content explicitly assessing depressive symptoms, a skill measure could simply reflect symptom severity. In our analyses of both patients at intake and controls, we found the relationship between the CCTS-SR and WOR remained significant even after partialling out depressive symptoms. Thus, the CCTS-SR exhibited concurrent validity not accounted for by any overlap with depressive symptoms.

In addition to our own efforts to develop a measure of CT skills, other groups have also developed CT skill measures, further highlighting the growing consensus for the need for such measures. Jarrett and colleagues (2011) recently reported on their development of the 8-item SoCT. Although we do not have data on these measures from the same sample, a few points on the comparison of these measures are worthy of note. Jarrett et al.’s SoCT has been validated in a large sample of depressed patients. Mid and post-treatment SoCT scores predicted odds of response to CT, controlling for initial depressive symptom severity (Jarrett et al., 2013). Factor analyses of our own measures and those of Jarrett et al. identified single factor solutions of CT skill measures. Thus, both of these studies suggest that CT patients’ mastery and performance of CT skills appears to be relatively well-characterized by an overall skill factor, with little need to distinguish among particular skills.

Limitations

We note four key limitations of this study. First, because therapists have no knowledge of their patients prior to treatment, the CCTS-TR requires patient contact before therapists are capable of rating patients’ CT skills. However, we have yet to determine how much patient contact is necessary for therapists to complete this assessment. We suspect the CCTS-TR could be utilized earlier in a course of treatment. Future research is needed to examine this issue. Second, our analyses examining the relation of CT skills and depressive symptoms involved only either relations at a given time point or concurrent changes in these variables over time. Our data did not allow us to examine changes in CCTS measured CT skills as a prospective predictor of changes in depressive symptoms or risk for relapse following treatment. Predicting risk of relapse remains an important issue for future research. Third, our sample size was not as large as would be desired for conducting exploratory factor analysis. We encourage additional attention to this issue in future research with larger samples. Finally, our data from control participants was only collected at one time point. While we have no reason to believe healthy control participants would have had higher or lower CCTS-SR scores on average on a subsequent occasion, our data do not allow us to address this issue.

Conclusion

In summary, this study examined two newly developed measures of CT skills: a self-report and therapist-rated version. Our findings suggest that these measures could prove useful in both research and clinical settings as short, easy to use alternatives to existing measures. Both measures were found to be associated with related measures of CT skills, and CT skill acquisition was associated with improvements in depressive symptoms over the course of CT. Future research should examine these measures as predictors of risk for relapse following treatment.

Appendix A. Factor Loadings of a Single Factor Solution of CCTS-SR Items among Depressed Patients at Intake and Post-Treatment

Factor Loading		CCTS-SR items
Intake	Post
.51	.74	1. Rather than letting a challenge overwhelm me, I imagined how to break the challenge down, developed a plan, and worked on it step-by- step.
.59	.71	2. At times when my mood was at its lowest, I stepped back and recognized that my self evaluations were probably overly negative.
.49	.76	3. I was aware of some specific patterns in negative thinking that have tended to affect the way I interpret new situations.
.56	.81	4. I made an effort to evaluate my negative thoughts by considering just the facts.
.62	.74	5. I examined evidence from my past or present to more carefully consider whether my negative thoughts are accurate or not.
.69	.67	6. When my negative thoughts and emotions really bothered me, I had a specific action plan of things I could do to cope.
.56	.81	7. When I had a negative emotional reaction, I noticed my negative thinking, and took time to evaluate my negative thoughts.
.47	.69	8. When I evaluated a situation as negative, I tried to think of how someone else would view the situation, and I used that to help me decide how to re-evaluate the situation myself.
.57	.84	9. I questioned my original negative thoughts and made an effort to develop alternative conclusions.
.50	.59	10. I recognized that negative feelings are related to negative thoughts I have about myself.
.65	.84	11. When I found myself upset about something, I took note of what I was thinking and worked to develop a more balanced view.
.63	.76	12. I recognized that beliefs I formed on the basis of past events and relationships may no longer be applicable in the same way today.
.47	.73	13. I have been recognizing that inaccurate, negative thoughts and judgments help to maintain my depression.
.56	.66	14. I have been aware of specific patterns in my negative thinking – beliefs that tend to fuel my negative emotions.
.46	.64	15. When I found myself worrying that something bad would happen, I reminded myself that the consequences might not be so terrible even if it did happen.
.60	.72	16. When I made an effort to correct my negative thinking, I was confident that my mood would get better.
.47	.62	17. When I was upset, I made an effort to engage in enjoyable activities that would be likely to improve my mood.
.41	.66	18. When I have felt down, I engaged in activities that were enjoyable or gave me a sense of accomplishment (to try to help my mood).
.59	.78	19. I have been confident that if I made an effort to be less pessimistic, my mood would improve.
.84	.86	20. I caught myself thinking negatively, recognized the negative bias, and re-evaluated the situation.
.72	.83	21. I often caught myself thinking in an irrational way and I actively worked to develop more rational views.
.51	.69	22. When I had a task that I might have had trouble undertaking, I made an effort to break the task up into smaller parts.
.61	.73	23. I took time to review specific thoughts I had during the most upsetting parts of my day.
.69	.66	24. When I blamed myself for something bad that happened, I took time to consider other factors that may have been involved.
.58	.75	25. I believed that working to more carefully evaluate my negative thoughts would probably help ease my depression.
.44	.72	26. I noticed specific automatic thoughts as they occurred.
.51	.68	27. Rather than avoiding a difficult decision, I weighed my options, developed a solution, and followed through with the decision I made.
.73	.85	28. When I got upset, I took time to step back from a situation and consider that my negative thoughts might be inaccurate.
.58	.68	29. When I had a tough interpersonal issue to address, I thought through how to be assertive in addressing the issue, tried to anticipate obstacles I might encounter, and initiated a conversation on the topic.

Note. CCTS-SR = Competencies of Cognitive Therapy Scale-Self Report. Clients were given the following prompt to read before completing the assessment:

• The following questions ask you about how much (if at all) you have used some specific, strategies to cope with negative moods in the last two weeks.
• There are several questions about ‘automatic thoughts’ and ways you may have reacted to these thoughts. To understand these questions, it is important to know what is meant by the term automatic thoughts. Automatic thoughts are thoughts that come to mind without much conscious effort throughout your day-to-day life. At times, such thoughts may occur without your taking any special notice of them. This questionnaire deals primarily with negative automatic thoughts. For example, someone who got passed over for a promotion at work might think “I must have made too many mistakes. I was a bad employee.”
• With the understanding of automatic thoughts described above, please use the following scale to indicate how well it describes your thoughts, beliefs and behaviors over the last two weeks.” Clients were asked to rate each item on a seven-point Likert-type scale ranging from “Not at all” to “Completely”. (Strunk, 2009, pp. 1).
• The CCTS-SR and the CCTS-TR may be reproduced and used free of charge for noncommercial research and in clinical practice if administered at no charge. Copies of the instruments must contain the following copyright notice: “Copyright 2009 Daniel R. Strunk, Ph.D., The Ohio State University, all rights reserved; reproduced with permission.”

Appendix B. Factor Loadings of a Single Factor Solution of CCTS-TR among Depressed Patients at Post-Treatment

Factor Loading	CCTS-TR Items
.92	Automatic Thoughts: Ability
.89	Automatic Thoughts: Frequency
.88	Automatic Thoughts: Independence
.82	Behavioral Activation: Ability
.82	Behavioral Activation: Frequency
.86	Behavioral Activation: Independence
.88	Core Beliefs: Ability
.90	Core Beliefs: Frequency
.89	Core Beliefs: Independence

Note. CCTS-TR = Competencies of Cognitive Therapy Scale-Therapist-Report. Therapists were given the following prompt:

The following questions ask you about your client and his or her ability, independent use and understanding of, and frequency of use for major components of Cognitive Therapy. Please rate your client across three domains (Behavioral Activation, Automatic Thoughts, and Core Beliefs) on three characteristics: their ability, independence, and frequency of using these strategies.

For each item, therapists were instructed to rate clients on a six-point Likert-type scale ranging from none to extensive.