Assessing Cognitive Therapy Skills Comprehension, Acquisition, and Use by Means of an Independent Observer Version of the Skills of Cognitive Therapy (SoCT-IO)

Abstract

Objectives

The purposes of this study were (1) to describe the adaptation and psychometric properties of the Skills for Cognitive Therapy (SoCT) measure for use by an independent observer (SoCT-IO) who rates the cognitive therapy (CT) skill acquisition, comprehension, and use by depressed adults and (2) to compare ratings of CT skill comprehension, acquisition, and use by independent observers to those by patients and therapists.

Method

Like the other SoCT versions, the SoCT-IO consists of 8 items that assess patients’ comprehension, acquisition, and use of cognitive and behavioral skills for managing depressive symptoms, using a 5-point Likert-type scale. Four experienced raters (two doctoral-level CT therapists and two bachelor-level non-therapists) used the SoCT-IO to rate 80 CT videotapes from both mid and later sessions in acute-phase CT from a randomized controlled trial for outpatients with recurrent major depression.

Results

The SoCT-IO ratings showed excellent internal consistency reliability and moderately high interrater reliability. Concurrent validity was demonstrated by convergence of the SoCT-IO with two other versions of the SoCT, one completed by therapists (SoCT-O) and the other by patients (SoCT-P). SoCT-IO ratings evidenced good predictive validity: Independent observers’ ratings of patient CT skills mid-phase in therapy predicted treatment response even when the predictive effects of SoCT ratings by therapists and patients were controlled.

Conclusions

The SoCT-IO is a psychometrically sound measure of CT skill comprehension, acquisition and use for rating outpatients with recurrent depression. The clinical utility and implications for using the SoCT-IO as a measure of CT skills acquisition are discussed.

Cognitive therapy (CT) is a structured, time-limited, present-focused psychotherapy that helps patients to learn and apply specific cognitive and behavioral strategies to alleviate or reduce symptom severity (e.g., Beck, Rush, Shaw, & Emery, 1979). Cognitive therapy has been shown to be efficacious for the treatment of mild, moderate, and severe depression (e.g., DeRubeis et al., 2005; Elkin et al., 1989; Tang, DeRubeis, Beberman, & Pham, 2005). The CT model proposes that negative automatic thoughts, dysfunctional core and intermediate beliefs, unhelpful coping strategies and information-processing biases are associated with depression. The primary aim of CT is to teach patients specific cognitive and behavioral skills and how to implement them to address problems and symptoms (Beck et al., 1979; Clark, Beck, & Alford, 1999).

The association between CT-skill acquisition or use and severity of depressive symptoms is poorly understood and is important for understanding patients’ progress during treatment. To examine this association, reliable and valid measures of CT skill acquisition and use are needed. In this regard, several studies have focused on measures of the frequency or independence of skill use such as the Behavioral Activation for Depression Scale (Kanter, Mulick, Busch, Berlin, & Martell, 2007), Cognitive-Behavioral Therapy Skills Questionnaire (Jacob, Christopher, & Neuhaus, 2011), Performance of Cognitive Therapy Strategies (Strunk, DeRubeis, Chiu, & Alvarez, 2007), and Skills of Cognitive Therapy (SoCT; Jarrett, Vittengl, Clark, & Thase, 2011). Others have focused on measures of the quality of skill use, such as the Ways of Responding (Barber & DeRubeis, 1992), Thought Diary Evaluation Form (Rees, McEvoy, & Nathan, 2005), Thought Record Skills Assessment (Neimeyer & Feixas, 1990), and Cognitive Therapy Awareness Scale (Wright et al., 2002). (For a review, see Hundt, Mignogna, Underhill, & Cully, 2013).

Most previous studies that have examined either the frequency or quality of skill use have used patient self-report. Although self-report questionnaires are convenient for practitioners and require little training to administer and interpret, ratings may be influenced by patients’ expectations of depressive-symptom change or social-desirability bias (Hundt et al., 2013). To address this concern, Jarrett and colleagues (2011) developed two versions of the Skills of Cognitive Therapy (SoCT) measure to assess patient comprehension, acquisition, and use of basic CT skills, one rated by patients (SoCT-P) and one by others (SoCT-O), using CT therapists in their first study. Results showed that both versions of the SoCT had excellent internal consistency reliability, as well as mild-to-moderate levels of concurrent and predictive validity (described in the Measures section).

The 2011 report used SoCT observer ratings by therapists who also provided the CT (Jarrett et al., 2011) and were highly correlated with patient reports. A potential limitation of using therapist ratings of CT skills is that therapists need to be available to complete these ratings during or shortly following the completion of the CT session in order to obtain the most valid ratings. Given this limitation, it would be useful to have an independent-observer measure that also includes behavioral anchors that describe how well the skills were understood or could be used by the patient independently of assistance from the therapist. Such a measure may be useful, especially when examining archival data, so that observers who did not provide treatment could review and rate previously recorded CT sessions. This expansion of the original SoCT scale would allow researchers the additional advantage of collaborating with raters who may or may not be highly trained in CT but who can learn to assess patient CT skills with manualized guidance, and can be calibrated on a standardized instrument to a desired level of inter-observer agreement. In addition, when therapists and patients are available for making ratings, then three different perspectives (i.e., patient, therapist, and independent observer) can be compared.

A few studies have used independent-observer ratings of CT skill acquisition. For example, the Performance of Cognitive Therapy Strategies (PCTS; Strunk et al., 2007) was created for independent observers to assess the extent to which patients demonstrated skills in each of three areas (behavioral activation, working with automatic thoughts, and schemas), through their performance in sessions, reports of use outside of sessions, and stated intentions to continue to use the skills. Strunk and associates found that the PCTS subscales of these three areas had moderately high interrater reliabilities (ICCs ranged from .74 to .83), had acceptable internal consistency reliabilities (alphas ranged from .74 to .94), and were significantly positively intercorrelated. The overall PCTS score—but not the subscales—significantly predicted relapse from depression. A recent study with depressed patients found that the PCTS were strongly associated with therapist and patient self-report versions (Strunk, Hollars, Adler, Goldstein, & Braun, 2014).

Few studies have examined all three perspectives—patient, therapist, and independent observer ratings—of CT skills. Thus, the purposes of this study were (1) to describe the development and psychometric properties of the Skills for Cognitive Therapy measure for use by an independent observer (SoCT-IO) who rates the cognitive therapy (CT) skill acquisition, comprehension, and use by depressed adults and (2) is to compare the extent to which ratings of CT skill comprehension, acquisition, and use by independent observers are similar to or different from ratings by patients and by therapists using the SoCT. To make this instrument available for multiple uses for research, we adapted the SoCT-O for use by independent observers (SoCT-IO) who may or may not have had prior CT training (Borman & Jarrett, 2010). Similar to the SoCT-P and SoCT-O versions, the SoCT-IO consists of eight items assessing skills comprehension, and acquisition and use. Unlike the previous versions of the SoCT, however, the SoCT-IO version assesses both the frequency and quality of CT skills. We evaluated the psychometric properties of the SoCT-IO measure using trained raters and previously recorded CT sessions. Specifically, we estimated the interrater and internal consistency reliabilities, the convergence of the SoCT-IO version with the patient and therapist observer versions, changes in skill levels over the middle and later phases of treatment for all three versions, and the concurrent and predictive validity of the SoCT-IO version in relation to depressive symptoms, cognitive content, and treatment response.

Method

We reviewed videotaped therapy sessions from a two-site clinical trial in which all patients were first treated with acute-phase CT (A-CT), after which responders were randomized to continuation-phase CT, fluoxetine, or pill placebo (called the Continuation Phase Cognitive Therapy Relapse Prevention [C-CT-RP] Trial; Jarrett & Thase, 2010). The acute phase consisted of 16-20 individual CT sessions (Beck et al., 1979) with the aim of reducing depressive symptoms and teaching specific CT skills. The first eight sessions occurred twice a week. Thereafter, “late responders,” patients who had obtained less than a 40% reduction in 17-item Hamilton Depression Rating Scale (HRSD-17) total scores compared to baseline, continued to receive twice-a-week CT sessions for 4 more weeks before beginning weekly sessions, whereas the remaining patients (“early responders”) began weekly sessions after week 4. Thus, patients could receive either 16 or 20 CT sessions by the end of week 12. See Jarrett and Thase (2010) for a full description of this clinical trial and Jarrett, Minhajuddin, Gershenfeld, Friedman, and Thase (2013) for the findings of the parent trial. For the purposes of this study, we selected videotapes from the middle and later sessions of the acute phase for which Cognitive Therapy Scale (CTS; Young & Beck, 1980) ratings had also been completed; tapes were selected without regard to acute phase outcome.

Participants

Patients were self- and clinician-referred to the Department of Psychiatry, Psychosocial Research and Depression Clinic at The University of Texas Southwestern Medical Center at Dallas and to the Mood Disorders Treatment Research Program at the Western Psychiatric Institute and Clinic of the University of Pittsburgh Medical Center. Recruitment methods included but were not limited to ads on the internet and in newspapers, churches, hospitals, clinics, and other community settings.

As previously described (Jarrett et al., 2011), sampled patients (N = 523) were outpatients with recurrent major depressive disorder (MDD) diagnosed using the Structured Clinical Interview for DSM–IV (SCID-I; First, Spitzer, Gibbon, & Williams, 1996) who (a) remitted between depressive episodes, had at least one prior episode with complete inter-episode recovery, or had antecedent dysthymic disorder and (b) had a score ≥ 14 on HRSD-17 (Hamilton, 1960; Williams, 1988). Excluded patients (a) had severe or poorly controlled concurrent medical disorders that may cause depression or require medication that could cause depressive symptoms; (b) had concurrent Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM–IV; American Psychiatric Association, 1994) psychotic or organic mental disorders, bipolar disorder, active alcohol or drug dependence, primary (i.e., predominant) obsessive compulsive disorder or eating disorders; (c) scored < 14 on the HRSD-17 at either the initial or a second interview; (d) could not complete questionnaires written in English; (e) were an active suicide risk; (f) had previously not responded to a trial of at least 8 weeks of CT conducted by a certified therapist; (g) had previously not responded to ≥ 6 weeks of 40 mg of fluoxetine; (h) were pregnant or planned to become pregnant during the first 11 months after intake; or (i) did not provide informed consent. Patients who were taking psychotropic medications were able to participate if they discontinued the medication, were not medicated for at least 1 week before entering the study, and agreed to maintain medication-free status unless randomized to the experimental continuation phase of the trial.

Of the 523 patients who consented, 410 completed acute phase CT; 395 patients attended the blinded evaluation used to define post-acute phase CT response. The current analyses included 74 (14.1%) patients with complete recordings of middle (38 recordings) and/or later CT (40 recordings) sessions (i.e., four patients contributed both middle and later session recordings).

Cognitive Therapists

As previously reported (Jarrett et al., 2011), CT was provided by 15 therapists (seven in Dallas; eight in Pittsburgh) consisting of 11 women and four men. At the Dallas site, all therapists were doctoral-level; at the Pittsburgh site, three were doctoral level, and the remaining therapists were master’s level. As defined by the site supervisors’ judgment and CTS scores ≥ 40, all therapists demonstrated proficiency in CT. Therapists also were trained to reliability in rating the HRSD-17 and making DSM–IV diagnoses of MDD. Experienced faculty members at each site led weekly group-supervision sessions and were available for individual case consultation. Ratings of CT competency, using the CTS, were completed by supervisors and peers. A research coordinator selected videotaped sessions approximately quarterly; all therapists were sampled.

Development of the Skills of Cognitive Therapy: Independent Observer (SoCT-IO) Version Items

The SoCT-IO is an 8-item, observer-rated measure that assesses patients’ comprehension, acquisition and use of basic cognitive therapy skills. The SoCT-IO is based on the items of the SoCT-O (Jarrett et al., 2011), which was designed to be used by experienced CT therapists (see below). Specifically, the SoCT-IO consists of one item that assesses understanding of the basic principles of CT and seven items that assess the frequency and the quality of various cognitive and behavioral skills using a 5-point Likert-type scale ranging from 1 (never) to 5 (always or when needed) with higher ratings indicating a higher frequency of use and greater acquisition of skills. The ratings for both the quality of skills as well as the frequency of skills are done by means of behavioral anchors that include criteria for each item (see Appendix).

When piloting the measure for the current study, it was determined that the interrater reliability of the scale could be improved by adding specific behavioral anchors for each rating point of each item. Although the SoCT-IO includes similar anchors to the SoCT-O and SoCT-P versions to assess the frequency of use, the SoCT-IO also includes behavioral anchors that describe the how well skills were understood or could be used by the patient independently of assistance from the therapist. For example, for the first SoCT-IO item (“The patient understood that his/her thoughts, feelings, and behaviors can contribute to his/her depression”), the response for a ‘5’ rating was augmented from simply “Always or When Needed” to “Always or When Needed. The patient displays or expresses recognition that his/her thoughts, feelings and behaviors definitely contribute to his/her depression. The patient sees the negative patterns and is becoming more adept at applying the strategies to help identify negative automatic thoughts, negative mood, and unhelpful behaviors and how these interact and contribute to his or her depression.” During the training phase for rating patients’ skills, a consensus meeting was held to review each of the items for all four raters. If ratings were divergent among the raters by more than 1 point, modifications were made to improve the specific responses for a given rating for each SoCT-IO item.

The SoCT-IO scale is scored by averaging the 8 items with possible scores ranging from 1 to 5, consistent with scoring the SoCT-P and SoCT-O. Higher scores reflect greater patient skill in applying cognitive therapy principles and coping strategies.

Measures

Skills of Cognitive Therapy: SoCT-P and SoCT-O Versions

The 8-item SoCT-P is a patient self-report measure that assesses patient comprehension, acquisition, and use of basic CT skills (Jarrett et al., 2011). Specifically, the SoCT-P consists of one item that assesses global understanding of the basic principles of CT and seven items that assess various cognitive and behavioral skills. In the previous study (Jarrett et al., 2011), patients completed the SoCT-P before a given therapy session and considered how often they had used a tool or skill in the past month. Ratings of patients’ skill usage are made on 5-point Likert-type scales ranging from 1 (never) to 5 (always or when needed); the scale is scored by averaging the 8 items. Higher scores reflect greater patient understanding of and skill in applying cognitive therapy principles and coping strategies. The SoCT-P has high internal consistency reliability, with alphas ranging from .86 to .89, and moderate correlations, ranging from .35 to .44, between with therapist ratings. The SoCT-O also is an 8-item scale that is completed by a therapist-observer to assess patients’ skill acquisition and frequency of use. As previously reported, after completing a study CT session, therapists rated the frequency of patients’ use of CT skills over the past month (Jarrett et al., 2011). The SoCT-O uses similar items and ratings to those of the SoCT-P. The SoCT-O had very high internal consistency reliability with alphas ranging from .90 to .93, and both SoCT-P and SoCT-O ratings showed good predictive validity for treatment response when collected at the midpoint and end of A-CT (Jarrett et al., 2011).

Cognitive Therapy Scale (CTS)

The CTS (Young & Beck, 1980) is an 11-item measure that assesses therapist competency in CT. Items are scored on a 7-point scale: 0 (Poor), 1 (Barely Adequate), 2 (Mediocre), 3 (Satisfactory), 4 (Good), 5 (Very Good), and 6 (Excellent). The 11 items are summed to indicate a CTS total score, ranging from 0 to 66. The CTS has been shown to have adequate internal and interrater reliability (Vallis, Shaw, & Dobson, 1986), with strong interrater agreement for general competence, and moderate agreement for specific items of the scale (Williams, Moorey, & Cobb, 1991). A CTS total score of 39 or lower is one standard deviation below the mean score of a group of certified cognitive therapists as rated on the CTRS (Shaw, 1984; Shaw et al., 1999). Therefore, traditionally, a CTS total score of 40 or higher has been used as the pre-determined standard for competency.

Major depressive episode (MDE)

The Major Depressive Disorder-Current section of the SCID-I was administered at the diagnostic evaluation, three times during A-CT, within 7 days of the last A-CT session, and any time a patient exited the protocol. As previously reported, agreement among raters on the SCID-I for MDE was found to be adequate (Jarrett et al., 2011).

Depression symptom severity

Patients reported depressive symptoms on the 21-item Beck Depression Inventory (BDI; Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) and 30-item Inventory of Depressive Symptomatology—Self-Report (IDS-SR; Rush, Gullion, Basco, Jarrett, & Trivedi, 1996), and clinicians administered the 17-item HRSD at the diagnostic evaluation and weekly during CT. These three measures mark the same depressive-symptom-severity construct in A-CT (Vittengl, Clark, Kraft, & Jarrett, 2005), and we aggregated them to form a robust symptom index for the current analyses. Specifically, we standardized the measures based on their distributions at intake and averaged measures taken in the middle (CT sessions 5-11; composite alpha = .97 based on n = 362 complete cases) and later (CT sessions 12-19; composite alpha = .98 based on n = 324 complete cases) in A-CT to correspond with the selection of session tapes to rate with the SoCT-IO.

Dysfunctional Attitudes Scale (DAS)

The DAS (Form A; Weissman, 1979) is a 40-item self-report measure of dysfunctional thoughts. Patients respond on 7-point Likert scales (ranging from “agree very much” to “disagree very much”) to statements about self-concept, happiness, perfectionism, and thoughts and feelings relevant to depression. Higher scores indicate more dysfunctional attitudes with greater severity. As previously reported, internal consistency was high (Jarrett et al., 2011).

Attributional Style Questionnaire (ASQ)

A revised, shortened version (Dykema, Bergbower, Doctora, & Peterson, 1996) of the original ASQ (Peterson et al., 1982) was used, in which patients generate causes for 12 hypothetical negative events and rate the extent to which the causes are stable (vs. unstable) and global (vs. specific). Respondents rate the 24 items (12 for each scale) from −3 to 3; higher total scores reflect more depressogenic (i.e., stable and global) attributions. In the previous study (Jarrett et al., 2011), the stable and global scales, respectively, demonstrated good internal consistency at the assessments used in the current analyses.

Beck Hopelessness Scale (BHS)

The BHS measures an individual’s negative expectancies about the future (Beck & Steer, 1988; Beck, Weissman, Lester, & Trexler, 1974). The scale consists of 20 true/false items; higher total scores mark greater hopelessness. The BHS demonstrated high internal consistency at the assessments used in the current analyses (Jarrett et al., 2011).

Self-Control Schedule

The SCS assesses use of self-control methods to solve behavioral problems (Rosenbaum, 1980) and may be viewed as a measure of learned resourcefulness. Respondents rate 36 items tapping use of cognitive strategies, problem-solving strategies, delay of gratification, and belief in one’s ability to regulate internal events on 6-point Likert-type scales ranging from +3 (very characteristic of me) to −3 (very uncharacteristic of me). The SCS had high internal consistencies at the assessments in the previous study (Jarrett et al., 2011).

Independent Ratings of Cognitive Therapy Skills using the SoCT-IO

Four SoCT-IO raters with expertise in rating CT sessions were recruited for this project, two from each site in Dallas and Philadelphia, respectively. The Philadelphia site included two doctoral-level psychologists trained in CT and the Dallas-site included two bachelor-level research assistants who read basic texts on CT as described in detail in the SoCT-IO training manual (Borman & Jarrett, 2010; included as an online supplement). This project consisted of a training phase and a formal-study phase. During the training phase, the SoCT-IO rating manual and scale (Borman & Jarrett, 2010) were reviewed and discussed by all raters with the original authors of the manual and scale. Raters independently reviewed 14 recordings of the same CT session (selected from a different CT trial) and rated each session independently. Subsequently, raters reviewed the ratings for each item during a weekly group meeting over approximately 3 months. Discrepancies in SoCT-IO item ratings were discussed and the final item scores were reached by consensus. All ratings were justified by observations made while reviewing the sessions. Subsequent modifications were made to the manual and response items for the SoCT-IO to provide additional clarification or explanation for rating specific items.

Raters were considered to be trained adequately when they met the following criteria: (a) the rater’s initial SoCT-IO summed total score was within a specific range (+/- 3 points) of the final score that was reached by group consensus, and (b) the initial SoCT-IO summed scores for 4 out of the last 5 tapes rated were within the +/- 3 points of the final SoCT-IO summed scores. Rater received feedback that they had completed the training successfully as soon as they met these criteria. However, during the training phase, each rater continued to rate tapes and to participate in the weekly consensus calls until all raters reached these criteria. Once all raters reached these criteria, the formal-study phase commenced¹.

During this phase, recordings that were not selected for review in the training phase were selected for review using three variables: phase of treatment (middle vs. later), treatment site (Pittsburgh vs. Dallas), and type of treatment response (response vs. no response). Response at the end of A-CT was defined as the absence of MDE and an HRSD-17 score ≤ 12, as assessed by an independent, blinded evaluator (Jarrett & Thase, 2010). Using these categories, a 2 × 2 × 2 contingency table was constructed and 10 tapes per cell were selected for a total of 80 tapes. Recordings were then selected for review if the session had been rated previously by other raters using the CTS.

The 80 recordings were randomly assigned to each of the four raters. Using a randomized block design, the assignment of recordings was stratified according to phase of acute therapy, site, and treatment response.

During the formal-rating phase, a recording that was not included in the pool of 80 tapes was selected for review each week. All raters rated this session independently. A consensus meeting was held each week and a final SoCT-IO score was determined by group consensus. If any individual rater did not have an initial SoCT-IO summed score that was within +/- 3 points of the final SoCT-IO score, then that rater’s recordings during the previous week were reviewed and rated by another rater who met the reliability criterion.

Among 80 tapes randomly selected, 78 were reviewed from 74 unique patients (i.e., 4 patients contributed two tapes each). One tape could not be rated fully due to an electronic malfunction during recording and was dropped from the analysis. In addition, one patient had two tapes from the middle of CT rated. For this patient, the tape from the ninth CT session was randomly selected for analysis and the tape from 11^th session was dropped. The remaining SoCT-IO video ratings from middle (n = 38) and later (n = 40) CT represented partly overlapping sets of patients (4 patients had both middle and later session tapes included).

In addition to the 78 tapes that were rated, 5 additional tapes were rated every week for 6 weeks by all 4 raters to check for calibration. During the formal rating phase, there were 3 occasions when 1 rater did not meet calibration criteria so a total 19 tapes that had been rated during the previous week were also rated by a second rater.

Management of Missing Data

All data were evaluated for missing items. Prorated SoCT-IO scores were calculated by averaging the item responses for those cases with one or two missing items. If more than two items were missing, then the SoCT-IO total score was determined to be missing. For those SoCT-IO tapes that required more than one observer rating, mean ratings of the SoCT-IO total scores were calculated for both raters and used in subsequent analyses. Based on the strong interrater reliability estimates during training and calibration, we used pooled data from the formal-rating phase for subsequent analyses testing the validity of the SoCT-IO ratings.

Results

We first estimated the reliability of the SoCT-IO ratings from 19 training and calibration tapes. Internal consistency reliability for the 8-item SoCT-IO ratings was very high (alpha = .93). Because each of the 4 raters provided SoCT-IO data on 19 patients, we computed a multilevel model (two levels, patients nested within raters) to estimate interrater reliability. In this model, the proportion of variance in SoCT-IO scores attributable to patients (and thus not attributable to disagreements among raters) was moderately high (intraclass correlation = .84). In addition, the four raters’ average SoCT-IO scores fell within a narrow range (2.51-2.57 on a 1-5 scale) and did not differ significantly from one another, F(3,53) = 0.29, p = .83. Thus, the four raters assigned patients SoCT-IO scores that were largely consistent in both rank order and absolute level, in support of their reliability, and there was no indication that SoCT-IO scores varied systematically by raters’ CT training (therapist vs. non-therapist) or site (Dallas vs. Pittsburgh).

We tested the convergence of the SoCT-IO ratings from the formal rating phase with the SoCT-P and SoCT-O ratings. As shown in Table 1, the SoCT-IO ratings correlated moderately with SoCT-O ratings (.32 mid- and .69 later in CT; 95% confidence intervals [-.03, .60] and [.46, .82], respectively) but somewhat less strongly with SoCT-P ratings (.09 mid- and .30 later in CT; 95% confidence intervals [-.27, .42] and [-.04, .58], respectively). Raw means for the SoCT-IO ratings during middle (n = 38, M = 2.68, SD = 0.64, range 1.50-4.00) and later (n = 40, M = 2.75, SD = 0.73, range 1.48-4.63) phases of CT.

Table 1.

Convergent Correlations of the SoCT-IO Ratings during Middle and Late Periods in Acute Phase CT

Convergent Measure	SoCT-IO Ratings Mid-CT			SoCT-IO Ratings Late CT
	n	r	95% CI	n	r	95% CI
Measures Including Therapist and Evaluator Reports
SoCT-O Therapist Ratings Mid-CT	33	.32*	-.03, .60	39	.63***	.38, .78
SoCT-O Therapist Ratings Late CT	31	.28	-.08, .58	38	.69***	.46, .82
Depressive Symptoms Mid-CT	38	-.36**	-.61, -.04	40	-.29*	-.55, .02
Depressive Symptoms Late CT	34	-.34**	-.61, .00	40	-.30*	-.56, .01
Response to CT (1 = yes, 0 = no)	38	.50***	.21, .70	40	.36**	.05, .60
Patient-Report Measures
SoCT-P Mid-CT	32	.09	-.27, .42	37	.25	-.09, .53
SoCT-P Late-CT	27	.22	-.18, .55	35	.30*	-.04, .58
Dysfunctional Attitudes Scale	32	-.06	-.40, .30	35	.07	-.27, .39
Beck Hopelessness Scale	32	-.36**	-.62, .00	36	-.44***	-.67, -.12
ASQ Stable Failure	30	-.49***	-.72, -.14	35	-.15	-.45, .20
ASQ Global Failure	30	-.36*	-.63, .01	35	-.03	-.36, .31
Self-Control Scale	32	.26	-.10, .56	36	.09	-.24, .41

Note. SoCT-IO = Skills of Cognitive Therapy-Independent Observer version. SoCT-P = Skills of Cognitive Therapy-Patient version. ASQ = Attributional Style Questionnaire. OR = odds ratio from logistic regression with response as the dichotomous outcome variable.

^*p < .05, one-tailed.

^**p < .05, two-tailed.

^***p < .01, two-tailed.

CI = confidence interval.

A repeated-measures multi-level model was conducted to compare independent ratings of CT skills with patient and therapist ratings of these skills (to examine main effects and rater-by-time interactions). Among patients with SoCT-IO ratings from the formal-rating phase, a repeated-measures multi-level model including SoCT-IO, SoCT-O, and SoCT-P ratings from the middle and later phases in CT showed significant main effects for time (middle or later in CT), F(1, 293) = 8.54, p < .01, and source (independent observer, therapist, or patient), F(2, 277) = 48.53, p < .01, but no significant time × source interaction, F(2, 287) = 1.75, p = .18. Estimated marginal means indicated that SoCT ratings increased a small amount from middle (3.09, SE = 0.07) to late (3.27, SE = 0.07) in CT, d = 0.24, p < .01, and that independent observers (2.73, SE = 0.08) rated skills substantially lower than did therapists (3.29, SE = 0.07), d = 0.70, p < .01, a medium effect size (ES), and especially patients (3.51, SE = 0.08), d = 0.98, p < .01, a large ES.

Among SoCT-IO ratings, 52 unique session tapes of therapists’ competencies were rated using the CTS. The correlation between the SoCT-IO and CTS ratings was moderate, r = .29, p = .04, 95% confidence interval (.02, .52). That is, therapists with greater competence tended to be associated with patients with greater skills as measured by the SoCT-IO.

We also considered the convergence of the SoCT-IO ratings with patient reports of depressive cognition. The degree of associations between SoCT-IO scores at middle and end of A-CT with the depression-symptoms aggregate and cognitive content (DAS, BHS, SCS, ASQ) at middle and end phases of A-CT was assessed using Pearson correlations. As shown in Table 1, the SoCT-IO ratings from mid-CT correlated significantly negatively with measures of hopelessness, stable failure attributions, and global failure attributions from mid-CT; SoCT-IO ratings from late-CT correlated significantly negatively with hopelessness and depressive symptoms, but not with failure attributions from late-CT. SoCT-IO ratings did not relate significantly to dysfunctional attitudes at either time point. The median convergent correlation of the SoCT-IO ratings with the measures of depressive cognition, after reflecting correlations to positive values to signify hypothesized relations and negative values for relations contradicting hypotheses, was .24 (range -.15 to .54), representing a small (.10-.30) effect size for validity coefficients (Cohen, 1988). In sum, patients’ behaviors demonstrating CT skills to independent observers related modestly to the patients’ self-reports of their depressive cognitive content. The correlations in Table 1 also suggest that the SoCT-IO ratings correlated somewhat more strongly and consistently with therapist and outside evaluator’ ratings of CT skills (median r = .35, range .28 to .69) compared to patient self-ratings of CT usage and skills (median r = .23, range -.15 to .54).

We next tested the SoCT-IO ratings’ prediction of depressive symptoms and response. As shown in Table 1, higher SoCT-IO ratings in the middle and later phases in CT correlated significantly with response. In parallel logistic regression models predicting response (see Table 2, models 1 and 5), one-point increments in SoCT-IO scores were associated with about 3 to 7 times greater odds of response to CT for SoCT-IO ratings from the late and middle phases of CT, respectively. For example, patients with SoCT-IO scores of 3 (using therapy skills “half the time”) vs. 2 (using therapy skills “almost never”) by the middle of CT would be expected to have 7 times greater odds of response by the end of CT.

Table 2.

Prediction of Response to Acute Phase CT from the SoCT-IO Ratings

	Beta	SE Beta	Odds Ratio	95% CI Odds Ratio
Model 1 (n = 38)
Intercept	-5.23	1.92
SoCT-IO Ratings Mid CT	1.96	0.71	7.12***	1.76-28.78
Model 2 (n = 32)
Intercept	-6.94	3.15
SoCT-P Ratings Mid CT	0.29	0.71	1.34	0.33-5.38
SoCT-IO Ratings Mid CT	2.25	0.86	9.50***	1.74-51.70
Model 3 (n = 33)
Intercept	-6.64	2.78
SoCT-O Ratings Mid CT	0.25	0.65	1.29	0.36-4.57
SoCT-IO Ratings Mid CT	2.23	0.90	9.33***	1.61-54.00
Model 4 (n = 38)
Intercept	-1.57	2.56
Depressive Symptoms Mid CT	-0.09	0.04	0.92**	0.84-0.998
SoCT-IO Ratings Mid CT	1.76	0.77	5.82**	1.28-26.42
Model 4b (n = 38)
Intercept	-0.22	2.92
Depressive Symptoms at Intake	-0.10	0.05	0.91**	0.82-0.998
SoCT-IO Ratings Mid CT	1.91	0.78	6.78**	1.48-31.06
Model 5 (n = 40)
Intercept	-3.05	1.45
SoCT-IO Ratings Late CT	1.11	0.52	3.04**	1.11-8.38
Model 6 (n = 35)
Intercept	-9.64	3.30
SoCT-P Late CT	2.13	0.74	8.43***	1.98-35.97
SoCT-IO Ratings Late CT	0.65	0.67	1.92	0.51-7.13
Model 7 (n = 38)
Intercept	-5.02	1.93
SoCT-O Ratings Late CT	1.64	0.73	5.18**	1.25-21.52
SoCT-IO Ratings Late CT	-0.15	0.76	0.86	0.20-3.82
Model 8 (n = 40)
Intercept	2.80	2.88
Depressive Symptoms Late CT	-0.23	0.08	0.79***	0.68-0.92
SoCT-IO Ratings Late CT	0.93	0.83	2.55	0.50-12.93
Model 8b (n = 40)
Intercept	-3.11	2.26
Depressive Symptoms at Intake	0.00	0.04	1.00	0.93-1.07
SoCT-IO Ratings Late CT	1.11	0.52	3.04**	1.10-8.39

Note. SoCT-IO = Skills of Cognitive Therapy-Independent Observer version. SoCT-P = Skills of Cognitive Therapy-Patient version, SoCT-O = Skills of Cognitive Therapy-Observer version.

^*p < .05, one-tailed.

^**p < .05, two-tailed.

^***p < .01, two-tailed.

CI = confidence interval.

The expected odds (and associated 95% confidence intervals) of response at the end of CT from selected SoCT-IO scores levels (i.e., 2.0, 2.25…5.0) at the middle and end of CT were estimated. Specific estimates of the probability of response at various SoCT-IO rating score levels are shown in Table 3. The greater dispersion of response probability estimates at mid-CT (e.g., 9% vs. 66% for scores of 1.5 vs. 3.0, respectively) compared to later in CT (e.g., 20% vs. 57% for scores of 1.5 and 3.0, respectively) reflects the stronger correlation between SoCT-IO video rating scores at mid-CT with response. For example, a score of 4.0 on the SoCT-IO ratings during the middle of CT predicted a 93% probability of response, whereas the same score of 4.0 later in CT was associated with a somewhat lower estimate of 80%.

Table 3.

Estimated Probability of Response (95% Confidence Interval) to Acute Phase CT at Selected SoCT IO Rating Score Levels

Score	Mid-CT	Later in CT
1.5	9 (2-37)%	20 (6-51)%
2.0	21 (8-46)%	30 (14-54)%
2.5	42 (25-61)%	43 (27-61)%
3.0	66 (44-82)%	57 (39-73)%
3.5	84 (56-95)%	70 (46-87)%
4.0	93 (64-99)%	80 (49-95)%
4.5	---out of observed range---	88 (51-98)%

Note. CT = Cognitive Therapy. SoCT-IO = Skills of Cognitive Therapy-Independent Observer version. Estimates and confidence intervals derived from logistic regression models.

Finally, we tested whether the SoCT-IO ratings uniquely predicted response when controlling SoCT-P ratings, SoCT-O ratings, or depressive symptoms. As shown in Table 2, the SoCT-IO ratings from the middle of CT predicted response beyond prediction from the SoCT-P (model 2), SoCT-O ratings (model 3), and depressive symptoms (model 4). In contrast, later in CT, the SoCT-IO ratings did not improve prediction over these other measures (models 6-8, respectively).

Discussion

The present study found that the SoCT-IO version was a psychometrically sound measure of CT skill comprehension, acquisition and use by outpatients with recurrent depression who were treated by proficient cognitive therapists. Specifically, results revealed that the SoCT-IO ratings had very high internal consistency reliability and moderately high interrater reliability. Concurrent validity was also found by convergence of the SoCT-IO ratings with the SoCT-P ratings and SoCT-O ratings. Interestingly, the SoCT-IO ratings were more strongly associated with therapist than patient ratings of skills, suggesting that observers see some aspect of patient skill that the patients miss or vice versa, or that observers of both kinds have somewhat different benchmarks for skill level (e.g., a manual for ratings of therapy). There was also a moderate association between the independent-observer ratings of patient skills with ratings of therapist skills that indicated that more competent CT therapists tended to have patients with higher CT skills; however, this reasonable hypothesis requires further study, given the small sample size. Moreover, independent-observer ratings of skill comprehension, acquisition and use were lower than study therapist ratings (medium ES) and were substantially lower than patient ratings (large ES). The lower ratings by uninvolved observers may reflect their independence or distance from the process. It is unknown which set of raters (involved vs. independent) are more accurate or valid.

The present study found evidence of concurrent validity as SoCT-IO scores were negatively associated with depressive symptoms and depressive cognition, and modestly and negatively associated with patient reports of hopelessness and failure attributions (though not dysfunctional attitudes). The magnitudes of these correlations reported in Table 1 are similar to the magnitudes of the correlations between the SoCT-P/SoCT-O and the depressive and cognitive content measures that were previously reported (Jarrett et al., 2011) which suggests that these measures have similar validity. In addition, the SoCT-IO ratings had some predictive validity for treatment response. Independent-observer ratings of CT skills at the middle and later phases of therapy were both associated with treatment response. Furthermore, independent-observer ratings of CT skills were predictive of treatment response, even when the predictive effects of SoCT-P and SoCT-O ratings were controlled. However, independent-observer ratings of CT skills during the later phase of treatment were not associated with treatment response when controlling for either of these variables. One possible explanation for this finding is that earlier skill use is more powerful therapeutically in a time-limited CT protocol. In addition, we found that the SoCT-IO ratings were predictive of treatment response even when depressive symptoms at intake and during the later phase of CT were controlled. This indicates that SoCT-IO ratings were uniquely predictive of treatment response and not simply associated with changes in depressive symptoms. Overall, the current data suggest that higher SoCT-IO ratings made during the middle of CT are likely associated with learning that occurred during therapy and may be more useful than ratings made later in CT. However, a pre-treatment measure of CT skills would be necessary to test this hypothesis.

The psychometric properties of the independent-observer ratings of the SoCT are similar to those previously reported for the observer (therapist) and patient ratings of the SoCT (Jarrett et al., 2011). However, there are several additional issues that warrant further investigation regarding potential differences among the three versions of the SoCT. Further research of the potential effect of social desirability and expectations of the raters is needed for all versions of the SoCT. For example, it is unclear whether SoCT-IO ratings may be more or less likely to be biased than patient and therapist SoCT ratings if the independent raters are blind to experiences that occur or other information that is available outside the therapy session. It is also possible that the SoCT-IO ratings may yield different findings when all contiguous sessions are rated versus when noncontiguous sessions are rated or when additional clinical information is provided to the independent observer that was not available to them in this study. Depending on the context or research hypotheses, one or more versions of the SoCT may be considered.

Finally, the SoCT-IO affords the opportunity for researchers to rate previously recorded sessions from other studies with archival data to ascertain the potential effects of skill comprehension, acquisition and use on depressive symptoms in CT. The SoCT-IO may be especially useful when researchers have previously recorded sessions and the therapists who conducted these sessions are no longer available to complete such ratings.

Several other limitations of the present study should be noted. First, as previously suggested, the SoCT-IO may need to be adapted for use in studies of other disorders (Jarrett et al., 2011). To date, the SoCT-IO has been used only for adult outpatients with recurrent depression. Second, the degree of associations between the SoCT-IO and the other rating scales used in the study should be interpreted with caution because these ratings were performed at different points in time. Specifically, both patients and study therapists were instructed to rate skills over the preceding month, whereas the independent observers were instructed to rate patient skills for a specific session. Another difference between the SoCT-IO and the other versions is noted in the anchors for individual items. For the patient and therapist-observer versions, the anchors focused exclusively on the frequency of skill use, whereas the behavioral anchors for the independent-observer version focused both on the frequency of use and the quality of the skill that was observed. Third, the present study is limited by only evaluating the ratings completed by two CT therapists at the Philadelphia site and two non-therapists at the Dallas site, so further research with additional therapists and non-therapists at other sites is needed to evaluate the generalizability of the findings. Finally, we note that therapists’ competency ratings of CT were provided mostly by the study supervisors, who could be biased toward rating therapists as more competent than ratings provided by independent observers, although this hypothesis also requires testing.

An important clinical utility of the SoCT-IO is the implication for improving the quality of supervision, including specific feedback to CT therapists that may be provided by an independent observer. The SoCT-IO has the potential for assisting in training other therapists in CT by sharpening the focus of the feedback on comprehension, acquisition and use of skills by patients. The SoCT-IO also may complement other supervision tools such as the CTS. Further research on the potential effects of using the SoCT for training purposes is needed.