Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Sep 1.
Published in final edited form as: Psychol Assess. 2011 Sep;23(3):578–586. doi: 10.1037/a0022485

Skills of Cognitive Therapy (SoCT): A New Measure of Patients’ Comprehension and Use

Robin B Jarrett 1, Jeffrey R Vittengl 2, Lee Anna Clark 3, Michael E Thase 4
PMCID: PMC3144321  NIHMSID: NIHMS277736  PMID: 21319902

Abstract

We describe the development and psychometric properties of a new measure called the Skills of Cognitive Therapy (SoCT) in depressed adults and their cognitive therapists. The eight-item SoCT assesses patients' understanding and use of basic cognitive therapy (CT) skills rated from the perspectives of both observers (SoCT-O; therapists in this report) and patients (SoCT-P). Ratings of patients’ skill usage are made on 5-point Likert-type scales ranging from 1 ("never") to 5 ("always or when needed”). Higher scores reflect greater patient skill in applying cognitive therapy principles and coping strategies. To develop this scale, we used a 33-item pool, rated by both patients and their therapists at the middle and end of CT (Ns = 359–416), and evaluated the reliability and concurrent and predictive validity of both versions of the scale. The SoCT has excellent internal consistency reliability and moderate correlations between the observer and patient versions. Importantly, the SoCT showed good predictive validity for response when collected at the midpoint of acute phase CT. Considering both patients’ self-ratings and clinicians’ SoCT ratings, the odds ratio for responding to CT was 2.6. We discuss the practical utility of the SoCT, as well as its theoretical importance in research of patient CT skills (e.g., acquisition, comprehension, and generalization) as putative moderators or mechanisms of symptom change in the therapy.

Keywords: cognitive therapy, depression, skill comprehension and use, assessment, mechanisms

Introduction

“Learning” is hypothesized to be a mechanism of change in most psychotherapy. While the theories underlying each school of psychotherapy differ with respect to what is taught and learned, most posit that learning promotes some form of change. If behavioral scientists can specify and measure what therapists teach and patients learn in psychotherapy, then investigators can begin to identify the key mechanisms facilitating or impeding change.

In cognitive and behavioral therapies for depression (CT, BT, and CBT), a primary goal is to teach patients to recognize and alter relations between emotions, thoughts, and behavior in order to relieve emotional distress and symptoms. A basic hypothesis underlying CT for depression is that the more skill patients show in acquiring, comprehending, using, and generalizing fundamental CT skills (which we label as general patient CT skill), the better the outcome of treatment will be (Jarrett, Vittengl, & Clark, 2008). Thus, the levels of patients’ CT skill comprehension and generalization are mechanisms hypothesized to promote the change process.

Currently, there are few measures available that attempt to assess all of the components of patients’ CT skill level (e.g., comprehension and usage). The Cognitive Therapy Awareness Scale (CTAS; Wright et al., 2002) is a 40-item multiple choice measure evaluating patients’ recognition of automatic thoughts and categorization of logical errors. Although the CTAS assesses comprehension of CT-relevant constructs, it does not assess the patient’s actual use or generalization of CT skills.

The Ways of Responding (WOR; Barber & DeRubeis, 1992; 2001) assesses patients’ “amount and quality” of compensatory skills (which is conceptualized as a type of coping skill) by applying content-analytic methods to rate respondent narratives. Patients are prompted to imagine themselves in problematic vignettes in which they are thinking particular negative thoughts. They then rate how vividly they can imagine the scenario and write down what they would do and think in these situations.

In contrast to the WOR, which appears to tap skill comprehension, Strunk, DeRubeis, Chiu, and Alvarez (2007) created the Performance of Cognitive Therapy Skills Scale (PCTS) to assess not only patients’ skill comprehension but also their usage or performance. Observers were trained to rate patients’ CT skill use across three domains: behavioral activation, work on automatic thoughts, and/or schema or core-beliefs. Thirty-five outpatient responders to CT for moderate to severe depression showed increased skill usage as assessed by mean PCTS scores by the end of acute phase CT. Interestingly, scores on the WOR and PCTS appeared uncorrelated, suggesting, in this small sample, that skill comprehension and usage may be unrelated. Because the PCTS ratings rely on videotape review and content-analytic methods by raters, the PCTS is impractical for most practitioners to use and likewise may be difficult for researchers to use in large samples.

Other investigators have developed measures of distinct constructs that have been hypothesized to relate to patient CT skill, making them candidates for moderators or mechanisms of symptom change during CT. Examples include pretreatment use of other coping skills as measured by the Self Help Inventory (SHI; Burns & Nolen-Hoeksma, 1991), measures of homework compliance (e.g., Callan et al., 2010; Gaynor, Lawrence, & Nelson-Gray, 2006; and Kazantzis, Deane, & Ronan, 2004), as well as therapist competence, often measured by the Cognitive Therapy Scale (CTS; Vallis, Shaw, & Dobson, 1986), or therapist adherence to CT, measured by the Collaborative Study Psychotherapy Rating Scale (CSPRS; Hollon, 1984; Webb, DeRubeis, & Barber, 2010). Each of these relevant constructs and instruments are important, but they do not directly assess the patients’ level of skill in understanding and applying CT fundamentals.

Thus, there appears to be an unmet need for a reliable, practical, and efficient measure of patients’ comprehension and use of CT tools and skills that will be valid for tracking and understanding processes of change in CT. To fill this gap, we developed a new eight-item measure, the Skills of Cognitive Therapy (SoCT), and describe its development, content, and psychometric properties here. The SoCT-Patient Version (SoCT-P) is a self-report questionnaire that reflects patients’ perception of their own skill level.1 The observer version of the SoCT (SoCT-O) is intended to reflect observers’ perception of patients’ skills and can be used by treating therapists, raters, or training supervisors who watch sessions, videotapes, or read transcripts. The same eight items are administered in both the SoCT-P and the SoCT-O, but the items are rephrased to apply to ratings by the patient or by an observer (i.e., here the therapist). In this article, we report on relations among the SoCT and measures of depressive symptoms and cognitive content over the course of acute phase CT. We report on both the SoCT-P and -O, with therapists as observers. In addition to examining the psychometric properties of the SoCT, we tested the hypothesis that greater patient CT skill at the midpoint of acute phase CT predicts improved depressive symptoms at the end of CT.

Method

Data were drawn from an ongoing, two-site clinical trial comparing acute phase CT responders randomized to continuation-phase CT, fluoxetine, or pill placebo (called the Continuation Phase Cognitive Therapy Relapse Prevention [C-CT-RP] Trial, [Jarrett & Thase, 2010]). In developing the SoCT, we used data collected from the diagnostic evaluation and acute phase of CT. Thus, here we summarize methods relevant to the current analyses. We emphasize that during acute phase CT protocol, patients were not prescribed antidepressant medications. We refer readers to Jarrett and Thase (2010) for additional detail regarding the context in which the SoCT was developed. All methods were approved annually by the local Institutional Review Boards (IRBs) and by the trial’s Data Safety and Monitoring Board (DSMB). All patients provided written informed consent.

Participants

Patients were both self- and practitioner-referred to the Department of Psychiatry, Psychosocial Research and Depression Clinic at The University of Texas Southwestern Medical Center at Dallas (Principal Investigator: Robin B. Jarrett, Ph.D.) and to the Mood Disorders Treatment Research Program at the Western Psychiatric Institute and Clinic of the University of Pittsburgh Medical Center (Principal Investigators: Michael E. Thase, M.D. and Edward Friedman, M.D.). Recruitment methods included ads on the Internet and in newspapers, churches, hospitals, clinics, and other community settings. Sampled patients (N = 523) were outpatients with recurrent MDD diagnosed by the Structured Clinical Interview for DSM-IV (SCID-I; First, Spitzer, Gibbon, & Williams, 1996) who: (a) remitted between depressive episodes, had at least one prior episode with complete inter-episode recovery, or had antecedent dysthymic disorder; and (b) had a score of ≥ 14 on the 17-item Hamilton Rating Scale for Depression (HRSD-17; Hamilton, 1960; Williams, 1988). Excluded patients: (a) had severe or poorly controlled concurrent medical disorders that may cause depression or require medication that could cause depressive symptoms; (b) had concurrent DSM-IV psychotic or organic mental disorders, bipolar disorder, active alcohol or drug dependence, primary (i.e., predominant) obsessive compulsive disorder or eating disorders; (c) scored < 14 on the 17-item HRSD at either the initial or a second interview; (d) could not complete questionnaires written in English; (e) were an active suicide risk; (f) had previously not responded to a trial of at least 8 weeks of CT conducted by a certified therapist; (g) had previously not responded to ≥ 6 weeks of 40 mg of fluoxetine; (h) were pregnant or planned to become pregnant during the first 11 months after intake; or (i) did not provide informed consent. When needed, we also collected a physical exam and laboratory data to make certain that patients were diagnostically eligible. Patients who were taking psychotropic medications were able to participate if they discontinued the medication, were unmedicated for at least one week before entering the study, and agreed to maintain medication-free status.

Of the 523 patients who consented to the study, 410 completed acute phase CT; 395 patients attended the blinded evaluation used to define post acute phase CT response (see below). The current analyses include patients for whom self- and therapist-reported SoCT data were available at the middle and end of CT (Ns ranged from 359 to 416).

Acute Phase Cognitive Therapy (CT)

Acute phase CT consisted of a 16–20 individual session protocol (Beck, Rush, Shaw, & Emery, 1979) with the aim of reducing depressive symptoms and teaching CT skills. Patients could complete sessions over a maximum of 14 weeks, to allow for missed appointments to be rescheduled. Sessions 1–8 occurred twice weekly. Thereafter, patients who experienced < 40% reduction in HRSD-17 scores (compared to HRSD-17 scores from the diagnostic assessment) were categorized as “late responders” and continued receiving twice weekly sessions for a total of 20 sessions. Patients who experienced ≥ 40% reduction in HRSD-17 scores were categorized as “early responders” and began receiving once weekly sessions for a total of 16 sessions. The four additional CT sessions were provided to give late responders a better chance of experiencing symptom reduction, which also would increase the likelihood that they would be eligible to continue into the next phase of the study relevant to preventing relapse. Patients were not paid for their participation. Sessions were videotaped. No pharmacotherapy was prescribed.

Cognitive Therapists

The 15 therapists (7 in Dallas; 8 in Pittsburgh) included 11 women and 4 men. At the Dallas site, all therapists were doctoral level and, at the Pittsburgh site, 3 were doctoral level; the remaining therapists were masters level. Before therapists treated protocol patients, they demonstrated proficiency in: (a) CT, as defined by the site supervisors’ judgment and by maintaining CTS scores ≥ 40; and (b) their clinician ratings for the HRSD-17 and DSM-IV diagnoses of Major Depressive Disorder (MDD). Experienced faculty members led the weekly group supervision sessions at each site and were available for individual case consultation. The supervisory groups watched and completed the CTS for randomly selected, videotaped sessions. Therapy supervisors and their teams used the CTS to provide feedback to therapists. The Principal Investigators and DSMB monitored the quality of the CTS scores across sites and time.

Measures

Major Depressive Episode (MDE)

The Current Major Depressive Disorder section of the SCID-I for DSM-IV was administered at the diagnostic evaluation, three times during acute phase CT, within 7 days of the last acute phase CT session, and any time a patient exited the protocol. Reliability between raters using the previous version of the SCID has been adequate (major depressive disorder kappa = .72; Riskind, Beck, Berchick, Brown, & Steer, 1987). In the current dataset, inter-rater reliability for diagnoses of major depressive episodes was moderate. In a sample of 39 patients rated by 4 to 21 clinicians each, the median kappa of all pairwise comparisons was .53 while the uncorrected percent agreement among raters was 88%.

Depression Symptom Severity

Symptom severity measures included the 17-item HRSD, 30-item Inventory of Depressive Symptomatology—Self Report (IDS-SR; Rush et al., 1986), and the 21-item Beck Depression Inventory (BDI; Beck, Ward, Mendelson, Mock, & Erbaugh, 1961). Clinicians administered the HRSD-17 at the diagnostic evaluation, follow-up interview, weekly during acute phase treatment, within 7 days of the last acute phase CT session, and any time a patient exited the protocol. Patients completed the BDI and IDS-SR at diagnostic evaluation, weekly during acute phase treatment, within 7 days of the last acute phase CT session, and any time they exited the protocol. Higher scores on the HRSD-17, IDS-SR, and BDI indicate more severe depressive symptoms, and the reliability and validity of the measures for assessing depression symptom severity is well established in clinical research (e.g., Vittengl, Clark, Kraft, & Jarrett, 2005). Past research indicates that these measures mark the same underlying construct during acute phase CT (Vittengl et al., 2005) and so can be aggregated to provide a robust symptom index. In the current dataset, we standardized measures based on the distribution of scores at the first acute phase CT session, computed the mean of the patient reports (BDI and IDS-SR), and averaged the patient-report mean with the clinician-rated HRSD-17. Alpha internal consistency reliability computed for the two-index, patient-clinician composite was high at the assessments used in the current analyses: session 1 (.80), session 11 (.90), and at the end of acute phase CT (.90).

Dysfunctional Attitudes Scale (DAS)

The DAS (Form A; Weissman, 1979) is a 40-item self-report measure of dysfunctional thoughts. Patients respond on 7-point Likert scales (ranging from “agree very much” to “disagree very much”) to statements about self-concept, happiness, perfectionism, and thoughts and feelings relevant to depression. Higher scores indicate more dysfunctional attitudes with greater severity. The validity of the DAS for assessing depressive cognition is supported by its ability to distinguish persons diagnosed with depression versus non-depressed controls (Otto et al., 2007; Nelson, Stern, & Cicchetti, 1992). Alpha internal consistency reliability was high at the assessments used in the current analyses: intake (.93), after acute phase CT session 11 (.94), and at the end of acute phase CT (.94).

Attributional Style Questionnaire (ASQ)

We used a revised, shortened version (Dykema, Bergbower, Doctora & Peterson, 1996) of the original ASQ (Peterson et al., 1982), in which patients generate causes for 12 hypothetical negative events and rate the extent to which the causes are stable (vs. unstable) and global (vs. specific). Respondents rate the 24 items (12 for each scale) from −3 to 3; higher total scores reflect more depressogenic (i.e., stable and global) attributions. Higher scores have been correlated with both acute and chronic depression (Riso et al., 2003) and lower than average rehabilitation in cardiac patients (Bennett & Elliott, 2005). The stable and global scales, respectively, demonstrated good internal consistency at the assessments used in the current analyses: intake (.81, .79), after acute phase CT session 11 (.85, .81), and at the end of acute phase CT (.86, .84).

Beck Hopelessness Scale (BHS)

The BHS measures an individual’s negative expectancies about the future (Beck & Steer, 1988; Beck, Weissman, Lester, & Trexler, 1974). The scale consists of 20 true/false items; higher total scores mark greater hopelessness. Higher scores on the BHS correlate with greater severity of depression (Beck, Kovacs, & Weissman, 1975) and suicidality (Beck, Steer, Kovacs, & Garrison, 1985). The BHS demonstrated high internal consistency at the assessments used in the current analyses: intake (.89), after acute phase CT session 11 (.90), and at the end of acute phase CT (.92).

Self-Control Schedule (SCS)

The SCS assesses use of self-control methods to solve behavioral problems (Rosenbaum, 1980) and may be viewed as a measure of learned resourcefulness. Respondents rate 36 items involving use of cognitive strategies, problem-solving strategies, delay of gratification, and belief in one’s ability to regulate internal events on 6-point Likert-type scales ranging from +3 (very characteristic of me) to −3 (very uncharacteristic of me). Higher SCS scores reflect resourcefulness and correlate with higher confidence (Akgun, 2004), lower scores on the BDI (Slessareva & Muraven, 2004), and response to CT among patients with more severe depressive symptoms (Burns, Rude, Simons, Bates, & Thase, 1994). The SCS demonstrated good internal consistency at the assessments used in the current analyses: intake (.85), after acute phase CT session 11 (.87), and at the end of acute phase CT (.89).

Development of the SoCT Item Pool

The first author (RBJ) drafted the original SoCT pool and instructions to the respondents. The 33-item pool (Jarrett & Kraft, 1998) was finalized after the cadre of cognitive therapists at The University of Texas Southwestern Medical Center at Dallas’s Psychosocial Research and Depression Clinic thoroughly evaluated and provided feedback which was used to revise the draft instrument as follows.

The expert cognitive therapists were asked to suggest revisions to the original item pool content–including both item wording and domain coverage—to ensure that the measure assessed patient skills they considered important in CT. Subsequently, the therapists rated the revised 33 items after watching up to four cognitive therapy sessions to verify that items addressed observable behaviors. Some items were reworded, but all 33 items were retained and no items were added, because the experts verified that after the rewordings, the item pool had excellent content validity in terms of addressing all the important skills taught in cognitive therapy.

Skills of Cognitive Therapy (SoCT)

Patients and CT therapists rated 33 candidate items reflecting patients' understanding and use of basic cognitive therapy skills from 1 ("never") to 5 ("always or when needed”). Patients and therapists completed the SoCT after acute phase CT session 11 (i.e., midpoint of acute phase) and again at the end of acute phase CT, using the time frame "over the last month.”

Patients completed the SoCT-P before a given therapy session started, whereas therapists completed the SoCT-O after the therapy session ended. Both raters considered how often the patient had used the tool or skill from CT over the “past month.” The interval of the “past month” was selected because: (a) we reasoned that CT skills would develop over a longer interval than days or, perhaps, weeks, and skill assessment needed to mirror skill development; (b) we wanted both patients and therapists to consider use of skills between therapy sessions; and (c) some CT skills might be used only intermittently, so a relatively long time period was needed to allow for the opportunity for all targeted skills to be used.

Acute phase CT response

Response at the end of acute phase CT was defined as the absence of MDE and an HRSD-17 score of ≤ 12, as assessed by an independent, blinded evaluator. Seventy-four percent (n=292/395) of patients were considered responders in this post-treatment, blinded evaluation.

Results

Selection of SoCT Scale Items

To inform item selection, we submitted the 33 candidate items to principal axis factor analyses using half the sample, randomly selected. We then verified the internal consistency reliability of the final scale in the second half of the sample. Separate factor analyses for patients’ and therapists’ ratings, at both the midpoint and end of acute phase CT, supported 1-factor solutions (see Table 1). In all analyses, the first factor was very large (accounting for 66–82% of the common variance), and all items loaded highly (> .50) on it, and inter-item correlations were moderate to high (medians = .42–.69).

Table 1.

Principal Axis Factor Analysis of SoCT Item Pool

33-Item Pool in Development Sample 8-Item Scale
Assessment Eigenvalues
for First and
Second Factors		 Loadings on
First Factor:
Median (Range)		 Inter-Item
Correlation:
Median (Range)		 Alpha in
Replication
Sample
Therapist (Mid-Acute Phase CT) 20.8; 2.4 .80 (.65–.87) .61 (.25–.96) .90
Therapist (End of Acute Phase CT) 23.1; 1.9 .85 (.72–.92) .69 (.36–.95) .93
Patient (Mid-Acute Phase CT) 14.3; 3.1 .65 (.52–.81) .42 (.12–.83) .86
Patient (End of Acute Phase CT) 16.7; 2.8 .73 (.60–.80) .49 (.24–.92) .89
Open in a new tab

Note. SoCT = Skills of Cognitive Therapy; CT = Cognitive Therapy.

Because the factor analysis indicated that any relatively small subset of the items would produce a unidimensional scale with high internal consistency reliability, final selection of items was completed with the goals of maximizing content validity and the possibility of future testing of cross-modal convergent validity, that is, via correlating the measure with observable behaviors. More specifically, we selected items (a) to ensure coverage of a wide range of important CT skills (e.g., understanding connections of mood to thoughts and dysfunctional assumptions; testing the validity of negative cognitive content; engaging in positive activities) and (b) with behavioral content that may be easier to observe and score reliably (e.g., in future studies of CT session video recordings). The eight selected items (see Appendix) consisted of one item assessing understanding of the important basic principles of CT and seven items assessing various CT tools and skills, and formed an internally consistent scale in the halves of both the therapist and patient samples that were reserved for replication (alpha range .89 to .93; see Table 1). We used this scale in all further analyses. The scale is scored by averaging the eight items, with possible total scores ranging from 1 to 5. Higher scores indicate greater patient CT skill usage and understanding.

Skill Level and Changes Over Time

Both patients and therapists, at the middle and end of acute phase CT, reported that the patients demonstrated CT skills between “half” and “most” of the time, on average (means between 3 and 4; see Table 2). A repeated-measures multilevel model showed that patients rated themselves higher in skill than did therapists (F[1,1137] = 29.50, p < .01, d = 0.17) and that SoCT scores increased from the middle to the end of acute phase CT (F[1,1145] = 73.95, p < .01, d = 0.28). The rater × time interaction, however, was not significant (F[1,1134] = 0.49, p = .48, d = 0.05), indicating that patients’ and therapists’ scores increased about equally during acute phase CT. Study site (Dallas vs. Pittsburgh) did not predict SoCT scores as a main effect or interaction with rater or time (ps > .05) and was excluded from the final model.

Table 2.

Observed SoCT Means by Rater and Assessment Time

Assessment Patient Therapist
Time N M SD N M SD
Mid-Acute Phase CT 396 3.33 0.66 416 3.21 0.70
End of Acute Phase CT 359 3.59 0.72 383 3.42 0.80
Open in a new tab

Note. SoCT = Skills of Cognitive Therapy; CT = Cognitive Therapy.

Congruence between therapists’ and patients’ ratings was moderate at both the middle (.43) and end (.44) of CT (see Table 3). Correlations from the middle to end of CT were moderately strong for both therapist (.72) and patient (.67) ratings, consistent with some individual differences in skill growth during the last half of acute phase treatment.

Table 3.

Test-Retest and Convergent Correlations Among SoCT Assessments

Assessment 1 2 3
1. Patient Mid-CT ---
2. Patient End of CT .67 ---
3. Therapist Mid-CT .43 .35 ---
4. Therapist End of CT .39 .44 .72
Open in a new tab

Note. Ns=344–384. SoCT = Skills of Cognitive Therapy; CT = Cognitive Therapy. All correlations p < .01.

Relations of SoCT Scores with Depressive Symptoms and Cognitive Content

As shown in Table 4, depressive symptoms and cognitive content assessed before CT were largely unrelated to SoCT ratings at the middle and end of CT. Thus, CT skill development was largely independent of initial levels of depressive symptoms and cognitions. The exception to this pattern was that greater self-reported learned resourcefulness (as measured by the Self-Control Schedule) predicted greater self-reported (but not therapist-reported) CT skill.

Table 4.

Correlations of Depressive Symptoms and Cognitive Content Measures with SoCT Assessments

SoCT Assessment
Measure Patient
Mid-Acute
Phase CT		 Therapist
Mid-Acute
Phase CT		 Patient
End Acute
Phase CT		 Therapist
End Acute
Phase CT
Early/Pre-CT
 Depression Symptoms Aggregate −.07 −.13 .05 −.07
 Dysfunctional Attitudes Scale −.08 −.02 −.10 −.09
 Beck Hopelessness Scale −.07 .01 −.09 −.05
 Self-Control Schedule .29* .03 .25* .06
 Attributional Style Quest. - Stable −.09 −.01 −.07 −.08
 Attributional Style Quest. - Global −.12 −.10 −.05 −.12
Mid-CT
 Depression Symptoms Aggregate −.38* −.41* −.26* −.27*
 Dysfunctional Attitudes Scale −.37* −.16* −.29* −.21*
 Beck Hopelessness Scale −.39* −.30* −.36* −.31*
 Self-Control Schedule .43* .22* .41* .23*
 Attributional Style Quest. - Stable −.25* −.17* −.25* −.15*
 Attributional Style Quest. - Global −.32* −.17* −.29* −.15*
End of CT
 Depression Symptoms Aggregate −.31* −.31* −.49* −.39*
 Response
   (no MDE and HRSD-17 ≤ 12)		 .17*
(OR=1.9*)		 .21*
(OR=2.1*)		 .35*
(OR=3.5*)		 .31*
(OR=2.6*)
 Dysfunctional Attitudes Scale −.32* −.14* −.45* −.22*
 Beck Hopelessness Scale −.33* −.25* −.48* −.35*
 Self-Control Schedule .40* .17* .55* .27*
 Attributional Style Quest. - Stable −.25* −.15* −.33* −.21*
 Attributional Style Quest. - Global −.25* −.18* −.28* −.16*
Open in a new tab

Note. Ns=336–416. Primary values are Pearson’s r. Symptoms aggregate is the mean of one clinician-rated report (17-item Hamilton Rating Scale for Depression) and the average of two patient-rated reports (Beck Depression Inventory; Inventory of Depressive Symptomatology-Self Report) standardized relative to the first CT session assessment.

SoCT = Skills of Cognitive Therapy; CT = Cognitive Therapy; MDE = Major Depressive Episode; OR = odds ratio from logistic regression with response as the dichotomous outcome variable.

*

p < .01.

By mid-CT, and again at the end of CT, the SoCT related widely to the depressive symptom and cognitive content measures. Patients with higher SoCT scores had lower depressive symptoms, greater odds of response, lower dysfunctional attitudes, less hopelessness, higher learned resourcefulness, and fewer stable and global attributions for negative events. Not surprisingly, correlations tended to be stronger at the same, compared to across, assessment times.

Therapist- and Patient-Rated SoCT: Independent and Aggregate Predictions

Both patient- and therapist-rated SoCT scores obtained at the middle and end of CT predicted depression and cognitive content at the end of CT (see Table 4). We explored patient- and therapist-rated SoCT scores’ independent and aggregate predictive power to understand further the overlap and divergence between the two skill-rating perspectives. As shown in Table 5, both patient and therapist SoCT scores contributed uniquely to prediction of depressive symptoms and response to CT. An index formed from the average of patient and therapist SoCT ratings (see Table 5) predicted depressive symptoms and CT response at the end of CT slightly better than did patient and therapist ratings alone (see Table 4).

Table 5.

Independent and Aggregate Prediction of Depressive Symptoms and Cognitive Content Measures from Patient and Therapist SoCT Ratings

Mid-CT SoCT Predicting
End of CT Outcomes		 End of CT SoCT Predicting
End of CT Outcomes
Model 1 Model 2 Model 1 Model 2
Outcome Measure at End of CT Patient
(controlling
Therapist)		 Therapist
(controlling
Patient)		 Patient and
Therapist
Average		 Patient
(controlling
Therapist)		 Therapist
(controlling
Patient)		 Patient and
Therapist
Average
Depression Symptoms Aggregate −.20* −.22* −.37* −.40* −.19* −.51*
Response
   (no MDE and HRSD-17 ≤ 12)		 .09
(OR=1.4)		 .18*
(OR=1.8*)		 .23*
(OR=2.6*)		 .27*
(OR=2.6*)		 .18*
(OR=1.8*)		 .38*
(OR=4.6*)
Dysfunctional Attitudes Scale −.32* .01 −.27* −.45* −.02 −.39*
Beck Hopelessness Scale −.26* −.15* −.34* −.41* −.16* −.48*
Self-Control Schedule .40* .00 .33* .52* .03 .47*
Attributional Style Quest. - Stable −.23* −.04 −.23* −.29* −.09 −.30*
Attributional Style Quest. - Global −.23* −.08 −.24* −.26* −.05 −.25*
Open in a new tab

Note. Ns=325–409. Primary values are standardized betas from multiple regression analyses. Symptoms aggregate is the mean of one clinician-rated report (17-item Hamilton Rating Scale for Depression) and the average of two patient-rated reports (Beck Depression Inventory; Inventory of Depressive Symptomatology-Self Report) standardized relative to the first CT session assessment.

OR = odds ratio from logistic regression with response as the dichotomous outcome variable; SoCT = Skills of Cognitive Therapy; CT = Cognitive Therapy; MDE = Major Depressive Episode.

*

p < .01

In contrast, patient SoCT ratings taken at the middle and end of CT largely dominated therapist SoCT ratings in predicting patient-reported cognitive content measures assessed at the end of CT (see Table 5). All 10 correlations between patient SoCT ratings at the middle and end of CT remained significant predictors of the cognitive content variables at the end of CT when controlling therapist SoCT ratings, whereas controlling patient SoCT ratings reduced 8 of 10 therapist SoCT predictions into the non-significant range. The two exceptions were smaller but still significant predictions of the hopelessness measure from the therapist-rated SoCT (at the middle and end of CT) when controlling patient ratings. Further, averaged therapist and patient SoCT ratings (see Table 5) were typically slightly weaker predictors of the cognitive content measures than were patient SoCT ratings alone (see Table 4).

Odds of Response to CT based on SoCT Scores

To clarify patient, therapist, and aggregate SoCT ratings’ prediction of CT response, we computed the expected odds of response at the end of CT from selected SoCT score levels at the middle and end of CT (see Table 6). We include confidence intervals to highlight the need for replication of our findings with this new instrument. Higher SoCT scores at both time points predicted greater odds of response most clearly, but other patterns were also noteworthy. For example, lower SoCT scores (≤ 2.25) at the middle of CT were less predictive of non-response than were comparably low scores at the end of CT. In addition, higher SoCT scores (≥ 4.50) were associated with high to very high odds, but did not guarantee response. Finally, averaging patients’ and therapists’ scores yielded predictions of somewhat lower response rates with low SoCT scores (≤ 2.25) and slightly higher response rates with high SoCT scores (≥ 4.50), compared to either patients’ or therapists’ scores alone. This pattern is consistent with the aggregate SoCT’s stronger prediction of depressive symptoms and response (see Tables 4 and 5).

Table 6.

Expected Odds of Response (95% Confidence Interval) to Acute Phase CT at Selected SoCT Score Levels

SoCT Response Odds Based on Mid-CT SoCT Response Odds Based on End of CT SoCT
Score Patient Therapist Aggregate Patient Therapist Aggregate
2.00 55 (44–66)% 46 (35–57)%
2.25 60 (49–69)% 60 (50–68)% 53 (42–64)% 39 (28–52)% 52 (42–61)% 35 (24–47)%
2.50 63 (55–71)% 64 (57–71)% 59 (50–67)% 47 (37–57)% 58 (49–65)% 44 (34–54)%
3.00 70 (65–75)% 72 (67–76)% 70 (65–75)% 62 (55–69)% 69 (63–74)% 63 (56–69)%
3.50 76 (71–81)% 79 (74–83)% 79 (74–83)% 75 (70–80)% 78 (73–82)% 78 (73–83)%
4.00 82 (75–87)% 84 (78–89)% 86 (80–90)% 85 (80–89)% 85 (80–89)% 89 (84–92)%
4.50 86 (78–91)% 88 (81–93)% 91 (84–95)% 91 (86–95)% 90 (85–94)% 94 (90–97)%
4.75 88 (79–93)% 94 (89–96)% 92 (87–96)% 96 (92–98)%
Open in a new tab

Note. CT = Cognitive Therapy; SoCT = Skills of Cognitive Therapy. Ranges are 95% confidence intervals derived from logistic regression models. Estimates not shown when fewer than 10 patients had a SoCT score on either extreme.

Discussion

The field continues to search for moderators and mechanisms of symptom change in cognitive therapy for depression and yet has limited technologies to measure some of the most important variables hypothesized to affect change, such as patient cognitive therapy skill level. We report on the development of the SoCT as a reliable and valid measure of patient acquisition, comprehension, and generalization of CT skill in a large sample (collected over an eight and a half-year period) from the perspectives of patients and their therapists. The SoCT fills a unique niche as a short measure with both self-report and observer versions and can reduce assessment burden, compared to existing alternatives. The observer version is designed for use by therapists or other reliable raters knowledgeable of cognitive therapy principles (e.g., trainees or supervisors who are trained to criterion on rating patient CT skills while watching videotaped sessions or reading transcripts).

Results from this large sample of depressed patients and their therapists suggested that on average patients used CT skills “most” to “half” of the time and that skills increased over the course of CT. Although therapists tended to rate patient skill levels lower than the patients themselves, the two types of ratings converged moderately. Skill development in CT appeared to increase independent of pretreatment levels of depressive symptoms and negative cognition, but, not surprisingly, self-reported skill was positively related to pre-treatment levels of patient-reported learned resourcefulness. Skills of Cognitive Therapy (SoCT) scores from both patients and therapists at the midpoint and end of CT predicted the probability of response to cognitive therapy, which may have important implications for improving clinical care. We reported parameters (i.e., SoCT-P and -O solely vs. in combination and distinct criterion scores) that influence the strength of the prediction.

The generalizability of these findings is limited by the characteristics of the patient sample (i.e., adults presenting with recurrent MDD) and by the characteristics of the therapists, (i.e., highly experienced CT therapists who participated in regular supervision). In future studies, it will be important to evaluate the SoCT in patients with other forms of depression, in therapists with less experience, and in diverse clinical settings. Even presuming that the promise of the SoCT is confirmed in these cases, it is likely that the SoCT will require adaptation for other disorders treated with CT. For example, the first item (i.e., “The patient [I] understood that his/her [my] thoughts, feelings, and behaviors can contribute to his/her [my] depression”) could be revised to include another primary symptom such as “anxiety.” Similar disorder-specific revisions may be needed in items 2 and 4 when the non-mood disorder is primary or comorbid with depression. Whether other content revisions that pertain to cognitive therapy skills will be needed is a matter of future research.

Alternate time frames for assessing skill use may be important. In this study the respondents rated use of CT skills “over the past month.” Other time frames, such as days or weeks, could be evaluated. In addition, the present analyses are silent with respect to how long it takes to develop skill or begin to use CT skills in the interval from intake to the midpoint of CT. We did not collect a baseline SoCT because we reasoned that skills take time (and sessions) to develop. This assumption should be evaluated in future studies.

Finally, in this preliminary study we did not develop the methods nor evaluate the psychometrics underlying SoCT-O when observers who are not the treating therapists provide the ratings. Thus, relations between observers’ perceptions of patient CT skill level to that of patients and their therapists are not documented here. However, to justify the inherent time and expense of such observer ratings of patient skill level from videotapes or transcripts, these ratings will need to provide additional insights that are not available from patients and/or therapists.

The data herein suggest that the patients’ learning and use of CT compensatory strategies (which we think of as “skill”) predict fewer symptoms at the end of acute phase CT for adult outpatients with recurrent major depressive disorder. This result supports the hypothesis that greater patient CT skill (measured at the midpoint) predicts improvements in depressive symptoms (measured at the end of CT). The SoCT could be used in clinical practice to identify patients with sub-optimal skill and to intervene with adjunctive therapeutic tools designed to boost patient learning. We offer the SoCT for consideration in (a) practice settings, (b) research on other psychiatric disorders, and (c) theory building and revision regarding the moderators and mechanisms responsible for change in cognitive therapy.

Acknowledgments

We are indebted to our research teams and our colleagues at The University of Texas Southwestern Medical Center at Dallas, the University of Pittsburgh (where Dr. Thase was located during patient accrual), and the University of Pennsylvania (Dr. Thase’s current affliation). We are grateful to Dolores Kraft, Ph.D. and the UT Southwestern cadre of cognitive therapists who assisted in developing the original SoCT item pool. We appreciate the assistance of Julie Kangas, B.A., Lauren Dunlap, B.A., Joanne Sanders, M.S., and Theresa Morgan, M.Phil. in preparing this manuscript and the reviewers for their helpful critiques.

This report was supported by Grants Number K24 MH001571, R01 MH58397, R01 MH69619 (to Robin B. Jarrett, Ph.D.) and R01 MH58356 and R01 MH69618 (to Michael E. Thase, M.D.) from the National Institute of Mental Health (NIMH). We appreciate the support of our NIMH Program Officer, Jane Pearson, Ph.D. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIMH or the National Institutes of Health. We also appreciate the careful review by members of the trial’s Data Safety and Monitoring Board.

Appendix

Skills of Cognitive Therapy – Observer [Patient] Version

© 2010 Robin B. Jarrett, Ph.D., The University of Texas Southwestern Medical Center at Dallas, all rights reserved2

Directions – For each statement below, circle the number that best indicates how often the patient [you] used tools or skills from cognitive therapy during the past month, where:

  1. Never

  2. Almost Never

  3. Half the Time

  4. Most of the Time

  5. Always or When Needed

1. The patient [I] understood that his/her [my] thoughts, feelings, and behaviors can contribute to his/her [my] depression. 1 2 3 4 5
2. The patient [I] examined his/her [my] underlying assumptions (or schema) and how they contributed to his/her [my] depression. 1 2 3 4 5
3. The patient [I] identified automatic negative thoughts and completed thought records. 1 2 3 4 5
4. The patient [I] scheduled and participated in activities which improved his/her [my] mood. 1 2 3 4 5
5. The patient [I] looked for alternative explanations when he/she [I] had negative thoughts. 1 2 3 4 5
6. The patient [I] weighed the evidence for and against negative thoughts. 1 2 3 4 5
7. The patient [I] tested negative automatic thoughts or beliefs by setting up experiments. 1 2 3 4 5
8. The patient [I] stated his/her [my] thoughts in ways that could be tested. 1 2 3 4 5
Open in a new tab

2The SoCT may be reproduced and used free-of-charge for non-commercial research and in clinical practice if administered at no charge to test respondents. All copies of the SoCT must contain the following copyright notice: © 2010 Robin B. Jarrett, Ph.D., The University of Texas Southwestern Medical Center at Dallas, all rights reserved; reproduced with permission.

Footnotes

Publisher's Disclaimer: The following manuscript is the final accepted manuscript. It has not been subjected to the final copyediting, fact-checking, and proofreading required for formal publication. It is not the definitive, publisher-authenticated version. The American Psychological Association and its Council of Editors disclaim any responsibility or liabilities for errors or omissions of this manuscript version, any version derived from this manuscript by NIH, or other third parties. The published version is available at www.apa.org/pubs/journals/pas

Financial Disclosure

During the past two years Dr. Thase has consulted with, served on advisory boards for, or received honoraria for talks from: AstraZeneca, Bristol-Myers Squibb Company, Eli Lilly and Company, Forest Laboratories, GlaxoSmithKline, Janssen Pharmaceutica, Lundbeck, MedAvante, Inc., Neuronetics, Inc., Novartis, Otsuka, Pamlab, Pfizer Pharmaceuticals, Schering-Plough, Shionogi, Shire US Inc., Supernus Pharmaceuticals, Takeda, Transcept Pharmaceuticals, and Wyeth Pharmaceuticals and he has received grant support from Eli Lilly and Company, Forest Laboratories, GlaxoSmithKline, Otsuka, and Sepracor, Inc., in addition to funding from the NIMH. He has equity holdings for MedAvante, Inc. and has received royalties from American Psychiatric Publishing, Inc. (APPI), Guilford Publications, Herald House, and W.W. Norton & Company, Inc. Two books currently promoted by the APPI specifically pertain to cognitive therapy. Dr. Thase also discloses that his spouse is an employee of Embryon, Inc. (formerly Cardinal Health and Advogent), which does business with several pharmaceutical companies that market medications used to treat depression.

Dr. Jarrett’s medical center receives the fees from the cognitive therapy she provides to patients. Dr. Jarrett is a paid consultant to the NIMH.

Lee Anna Clark is now at the Department of Psychology, University of Notre Dame.

1

As explained subsequently, the final measure contains one item directly assessing understanding of CT principles and seven items assessing use of CT tools or skills. Because understanding is implicit in the use of CT tools/skills, for simplicity’s sake, hereafter, we refer primarily to the use or application of these skills, rather than always repeating “understanding” as well.

Contributor Information

Robin B. Jarrett, Department of Psychiatry, The University of Texas Southwestern Medical Center at Dallas

Jeffrey R. Vittengl, Department of Psychology, Truman State University

Lee Anna Clark, Department of Psychology, University of Iowa.

Michael E. Thase, Department of Psychiatry, The University of Pennsylvania School of Medicine and Department of Psychiatry, The University of Pittsburgh Medical Center

References

  1. Akgun S. The effects of situation and learned resourcefulness on coping responses. Social Behavior and Personality. 2004;32:441–448. [Google Scholar]
  2. Barber JP, DeRubeis RJ. The Ways of Responding: A scale to assess compensatory skills taught in cognitive therapy. Behavioral Assessment. 1992;14:93–115. [Google Scholar]
  3. Barber JP, DeRubeis RJ. Change in compensatory skills in cognitive therapy for depression. Journal of Psychotherapy Practice and Research. 2001;10(1):8–13. [PMC free article] [PubMed] [Google Scholar]
  4. Beck AT, Kovacs M, Weissman A. Hopelessness and suicidal behavior: An overview. Journal of the American Medical Association. 1975;234:1146–1149. [PubMed] [Google Scholar]
  5. Beck AT, Rush AJ, Shaw BF, Emery G. Cognitive Therapy of Depression. New York: Guilford Press; 1979. [Google Scholar]
  6. Beck AT, Steer RA. Manual for the Beck Hopelessness Scale. San Antonio, TX: The Psychological Corporation; 1988. [Google Scholar]
  7. Beck AT, Steer RA, Kovacs M, Garrison BS. Hopelessness and eventual suicide: A 10-year prospective study of patients hospitalized with suicidal ideation. American Journal of Psychiatry. 1985;142:559–563. doi: 10.1176/ajp.142.5.559. [DOI] [PubMed] [Google Scholar]
  8. Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Archives of General Psychiatry. 1961;4:561–571. doi: 10.1001/archpsyc.1961.01710120031004. [DOI] [PubMed] [Google Scholar]
  9. Beck AT, Weissman A, Lester D, Trexler L. The measurement of pessimism: The Hopelessness Scale. Journal of Consulting & Clinical Psychology. 1974;42:861–865. doi: 10.1037/h0037562. [DOI] [PubMed] [Google Scholar]
  10. Bennett KK, Elliott M. Pessimistic explanatory style and cardiac health: What is the relation and the mechanism that links them? Basic and Applied Social Psychology. 2005;27:239–248. [Google Scholar]
  11. Burns D, Nolen-Hoeksema S. Coping styles, homework compliance, and the effectiveness of cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology. 1991;59(2):305–311. doi: 10.1037//0022-006x.59.2.305. [DOI] [PubMed] [Google Scholar]
  12. Burns DD, Rude S, Simons AD, Bates MA, Thase ME. Does learned resourcefulness predict the response to cognitive behavioral therapy for depression? Cognitive Therapy and Research. 1994;18:277–291. [Google Scholar]
  13. Callan JA, Dunbar-Jacob J, Sereika SM, Stone C, Fasiczka AL, Jarrett RB, Thase ME. Barriers to Cognitive Behavioral Therapy Homework Completion Scale- Depression Version: Development and psychometric evaluation. 2010 doi: 10.1521/ijct.2012.5.2.219. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Dykema J, Bergbower K, Doctora JD, Peterson C. An Attributional Style Questionnaire for general use. Journal of Psychoeducational Assessment. 1996;14:100–108. [Google Scholar]
  15. First MB, Spitzer RL, Gibbon M, Williams JB. Structured Clinical Interview for DSM-IV Axis I Disorders-Patient Edition (SCID-I/P, Version 2.0) New York: New York State Psychiatric Institute, Biometrics Research Department; 1996. [Google Scholar]
  16. Gaynor S, Lawrence S, Nelson-Gray RO. Measuring homework compliance in cognitive-behavioral therapy for adolescent depression: Review, preliminary findings, and implications for theory and practice. Behavior Modification. 2006;30(5):647–672. doi: 10.1177/0145445504272979. [DOI] [PubMed] [Google Scholar]
  17. Hamilton M. A rating scale for depression. Journal of Neurology, Neurosurgery & Psychiatry. 1960;23:56–62. doi: 10.1136/jnnp.23.1.56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Hollon SD, Evans MD, Elkin I, Lowery A. System for rating therapies for depression. Paper presented at the 92nd Annual Convention of the American Psychological Association; Toronto, Ontario, Canada. 1984. Aug, [Google Scholar]
  19. Jarrett RB, Kraft D. Cognitive Therapy Skill Rating Scale - Patient and Therapist Versions. 1998 Unpublished instruments. [Google Scholar]
  20. Jarrett RB, Thase ME. Comparative efficacy and durability of continuation phase cognitive therapy for preventing recurrent depression: Design of a double-blinded fluoxetine- and pill placebo-controlled randomized trial with 2-year follow-up. Contemporary Clinical Trials. 2010;31(4):355–377. doi: 10.1016/j.cct.2010.04.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Jarrett RB, Vittengl JR, Clark LA. Preventing recurrent depression. In: Whisman MA, editor. Adapting cognitive therapy for depression: Managing complexity and comorbidity. New York: Guilford Publications; 2008. [Google Scholar]
  22. Kazantzis N, Deane FP, Ronan KR. Assessing compliance with homework assignments: Review and recommendations for clinical practice. Journal of Clinical Psychology. 2004;60(6):627–641. doi: 10.1002/jclp.10239. [DOI] [PubMed] [Google Scholar]
  23. Nelson LD, Stern SL, Ciccheti DV. The Dysfunctional Attitudes Scale: How well can this measure depressive thinking? Journal of Psychopathology and Behavioral Assessment. 1992;14:217–224. [Google Scholar]
  24. Otto MW, Teachman BA, Cohen LS, Cohen CN, Vitonis AF, Harlow BL. Dysfunctional attitudes and episodes of major depression: Predictive validity and temporal stability in never-depressed, depressed, and recovered women. Journal of Abnormal Psychology. 2007;116:475–483. doi: 10.1037/0021-843X.116.3.475. [DOI] [PubMed] [Google Scholar]
  25. Peterson C, Semmel A, von Baeyer C, Abramson LY, Matalsky GI, Seligman MP. The Attributional Style Questionnaire. Cognitive Therapy Research. 1982;6:287–300. [Google Scholar]
  26. Riskind JH, Beck AT, Berchick RJ, Brown G, Steer RA. Reliability of the DSM-III diagnoses for major depression and generalized anxiety disorder using the Structured Clinical Interview for DSM-III. Archives of General Psychiatry. 1987;44:817–820. doi: 10.1001/archpsyc.1987.01800210065010. [DOI] [PubMed] [Google Scholar]
  27. Riso LP, du Toit PL, Blandino JA, Penna S, Dacey S, Duin JS, …Ulmer CS. Cognitive aspects of chronic depression. Journal of Abnormal Psychology. 2003;112:72–80. [PubMed] [Google Scholar]
  28. Rosenbaum M. A schedule for assessing self-control behaviors: Preliminary findings. Behavior Therapy. 1980;11:109–121. [Google Scholar]
  29. Rush AJ, Giles DE, Schlesser MA, Fulton CL, Weissenburger JE, Burns CT. The Inventory for Depressive Symptomatology (IDS): Preliminary findings. Psychiatry Research. 1986;18:65–87. doi: 10.1016/0165-1781(86)90060-0. [DOI] [PubMed] [Google Scholar]
  30. Slessareva E, Muraven M. Sensitivity to punishment and self-control: The mediating role of emotion. Personality and Individual Differences. 2004;36:307–319. [Google Scholar]
  31. Strunk DR, DeRubeis RJ, Chiu AW, Alvarez J. Patients’ competence in and performance of cognitive therapy skills: Relation to the reduction of relapse risk following treatment for depression. Journal of Consulting and Clinical Psychology. 2007;75(4):523–530. doi: 10.1037/0022-006X.75.4.523. [DOI] [PubMed] [Google Scholar]
  32. Vallis T, Shaw BF, Dobson KD. The Cognitive Therapy Scale: Psychometric properties. Journal of Consulting & Clinical Psychology. 1986;54:381–385. doi: 10.1037//0022-006x.54.3.381. [DOI] [PubMed] [Google Scholar]
  33. Vittengl JR, Clark LA, Kraft D, Jarrett RB. Multiple measures, methods, and moments: A factor-analytic investigation of change in depressive symptoms during acute phase cognitive therapy. Psychological Medicine. 2005;35:693–704. doi: 10.1017/s0033291704004143. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Webb CA, DeRubeis RJ, Barber JP. Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology. 2010;78(2):200–211. doi: 10.1037/a0018912. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Weissman AN. Dissertation Abstracts International. Vol. 40. 1979. The Dysfunctional Attitudes Scale: A validation study (Doctoral dissertation, University of Pennsylvania, 1979) pp. 1389B–1390B. [Google Scholar]
  36. Williams JB. A structured interview guide for the Hamilton Depression Rating Scale. Archives of General Psychiatry. 1988;45:742–747. doi: 10.1001/archpsyc.1988.01800320058007. [DOI] [PubMed] [Google Scholar]
  37. Wright JH, Wright AS, Salmon P, Beck AT, Kuykendall J, Goldsmith LJ, Zickel MB. Development and initial testing of a multimedia program for computer-assisted cognitive therapy. American Journal of Psychotherapy. 2002;56(2):76–86. doi: 10.1176/appi.psychotherapy.2002.56.1.76. [DOI] [PubMed] [Google Scholar]

RESOURCES