Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Dec 30.
Published in final edited form as: Psychiatry Res. 2015 Oct 30;230(3):819–825. doi: 10.1016/j.psychres.2015.10.033

Creation and Validation of the Cognitive and Behavioral Response to Stress Scale in a Depression Trial

Adam S Miner a, Stephen M Schueller b, Emily G Lattie b, David C Mohr b,*
PMCID: PMC4681670  NIHMSID: NIHMS736475  PMID: 26553147

Abstract

The Cognitive and Behavioral Response to Stress Scale (CB-RSS) is a self-report measure of the use and helpfulness of several cognitive and behavioral skills. Unlike other measures that focus on language specific to terms used in therapy, the CB-RSS was intended to tap the strategies in ways that might be understandable to those who had not undergone therapy. The measure was included in a clinical trial of cognitive-behavioral therapy for depression and completed by 325 participants at baseline and end of treatment (18 weeks). Psychometric properties of the scale were assessed through iterative exploratory and confirmatory factor analyses. These analyses identified two subscales, cognitive and behavioral skills, each with high reliability. Validity was addressed by investigating relationships with depression symptoms, positive affect, perceived stress, and coping self-efficacy. End of treatment scores predicted changes in all outcomes, with the largest relationships between baseline CB-RSS scales and coping self-efficacy. These findings suggest that the CB-RSS is a useful tool to measure cognitive and behavioral skills both at baseline (prior to treatment) as well as during the course of treatment. Keywords: Development, Validation, Telehealth

1. Introduction

Cognitive behavioral therapy (CBT) is an established, evidence-based treatment for depression (American Psychiatric Association, 2010; Hollon and Dimidjian, 2014). A major focus of CBT is educating patients about the role that thoughts and behaviors play in generating and maintaining depressive symptoms and teaching them cognitive and behavioral skills that, when applied in their lives, facilitate therapeutic gains. Indeed, acquisition of these skills is proposed to be a critical mechanism of change in CBT for depression (Barber and DeRubeis, 1989). As the use of these skills predominantly takes place outside of the therapy session, it is important to develop measures to assess if and how they are used. Measures of cognitive and behavioral skills could aid the investigation of mechanisms of change and serve as an outcome that is more proximal than depressive symptoms in outcome studies.

Measures of cognitive and behavioral skills overlap with several research areas. First, many self-report scales measure general coping skills. Examples include The Ways of Coping Checklist (Folkman and Lazarus, 1980) which presents a series of Yes/No questions that measure problem focused and emotion-focused coping strategies and the COPE Inventory (Carver et al., 1989) and Coping Skills Questionnaire (CSQ; Rosenstiel and Keefe, 1983) which both assess a broader range of coping strategies based on the frequency that a person applies each strategy. Second, several measures have been develop that measure skills more specific to CBT. While the constructs and methods vary from measure to measure, studies using these measures have found that frequency and quality of patient CBT skill use relate to depressive symptoms. For example, using the patient version and the observer (therapist) version of the Skills of Cognitive Therapy Scale (SoCT-P and SoCT-O), Jarrett and colleagues (2011) found that patients receiving cognitive therapy who reported higher comprehension and use of cognitive and behavioral skills at mid and post-treatment had lower depressive symptoms post-treatment. The Cognitive-Behavioral Therapy Skills Questionnaire (CBT-SQ; Jacobs et al., 2011) assesses frequency of both behavioral and cognitive skill use, and has been shown to improve among patients receiving CBT-oriented therapy. Behavioral skill use frequency has been assessed by the Behavioral Activation and Depression Scale (BADS; Kanter et al., 2007), in which patients report how frequently they engaged in behaviors related to depression such as doing things to avoid feeling sad or engaging in distracting activities. The scale also measures vulnerabilities and problematic coping strategies. In a trial for atypical depression, BADS scores improved concurrently with decreases in depressive symptoms (Weinstock et al., 2011).

Existing measures of cognitive and behavioral skills often use language that is specific to, and taught during, therapy. This may introduce artifacts in pre-post assessment such as response shift (Sprangers and Schwartz, 1999), in which changes in self-reported skills may reflect not only changes in the use of the skills, but also changes in the patient’s understanding of the construct (e.g. the definition of terms like negative thought, positive activity) and in the scaling (e.g. the meaning of what frequent use of a skill means may be change through CBT, thereby complicating the interpretation of change in scores. A notable exception is the Competencies of Cognitive Therapy Scale (Strunk et al., 2014), which uses everyday language meant to be understood without exposure to cognitive therapy, and can be used as a pre-post measure of cognitive skills. When people enter treatment they may have some patterns of responding to stressful situations, and some of these patterns may be adaptive and even consistent with cognitive and behavioral principles. Even outside of the context of treatment, individuals with better coping skills demonstrate fewer increases in depressive symptoms in the face of stressful life events (Adler et al., 2013), and so one’s responses when faced with stressors may be valuable to assess at the beginning of treatment.

Developing useful self-report measures is valuable because other approaches to evaluating the quality of skills learned during therapy can be time consuming. The Skills of Cognitive Therapy – Independent Observer version (SoCT-IO; Brown et al., 2015) is an observer-rated measure that rates the frequency and quality of skill use during the review of taped therapy sessions. Observer ratings can provide a unique and valuable perspective. The SoCT-IO has demonstrated the ability to predict response to cognitive therapy when used to evaluate mid- to late- sessions during a course of cognitive therapy. However, it may be time- and cost-prohibitive for many researchers and clinicians to train and employ independent observers to rate therapy sessions on this scale. Another strategy to assess use of skills has been through independent coding of homework assignments, such as thought records (e.g., Rees et al., 2005). Although this has the benefit of addressing situations relevant to individual patients, it is also resource intensive, requiring both time and training, and can only be used with people who have learned how to complete the specific homework assignment.

Self-report scales measuring coping, whether general or specific to CBT tend evaluate the use of coping strategies and skills based on frequency. It has been argued that such frequency-based measures are weak, as the usefulness of coping is dependent upon a good fit between a stressor and the coping strategy (Vitaliano et al., 1990). Any given strategy may be helpful in one context, but not in others. While evaluating the fit between coping and stressor can be useful, it is also time intensive. One proposed way to get closer to “fit” has been to evaluate self-reported self-efficacy and usefulness of coping strategies (Chesney et al., 2006). Indeed, such measures have been shown to be strongly predictive of response to CBT (Stiles-Shields et al., 2015).

Self-report measures of cognitive and behavioral skills that are relevant prior to and during treatment can advance the field by improving the understanding of how acquisition of skills and skill use during treatment relates to benefits accrued from CBT. Because frequency of skills use may not reflect an appropriate or adaptive use of skills (e.g. a person may use a skill in contexts where it is not useful), perceived usefulness ratings may add a unique perspective on self-report of skills (Chesney et al., 2006). The aim of this study was to develop a brief self-report measure that identifies cognitive and behavioral skills used and their perceived helpfulness for patients in CBT treatment, and to evaluate that measure in the context of CBT.

2. Methods

2.1. Participants

The sample comes from a randomized controlled trial of 325 participants receiving either face-to-face CBT or telephone administered CBT. Trial details and main outcomes can be found in the primary outcome paper (Mohr et al., 2012). Briefly, CBT was administered in an identical manner, with the exception of medium (face-to-face vs. telephone) and included both behavioral activation and cognitive restructuring strategies. There were no differences between treatments at post-treatment on measures of depressive symptoms. As such, for the current analysis, participants from both conditions were combined. Participants were required to meet criteria for major depressive disorder and score 16 or higher on the Hamilton Depression Rating Scale (HAM-D, Hamilton, 1960). Participants were also required to be at least 18 years old, to speak and read English, and to be available for 18 sessions of face-to-face or telephone therapy. Individuals were excluded if they (a) met diagnostic criteria for dementia, a severe psychiatric disorder, or depression of organic etiology; (b) had hearing or visual impairment that would prevent study participation; (c) reported alcohol or substance use severe enough to disrupt treatment as judged by two study psychologists; (d) exhibited severe suicidality; (e) were receiving or planning to start individual psychotherapy; or (e) had started antidepressant medication in the previous 10 days.

The study sample had a mean age of 47.5 years and a standard deviation of 13.1 years. The sample was 78% female (n = 252) and 14% Hispanic or Latino (n = 44), with 2 participants declining to answer. Participants were 24.2% Black (n = 72), 62.8% White (n = 187), 10.1% reporting more than one race (n = 30), and 3% other (n = 9), with 27 participants not reporting race. Married or cohabitating participants made up 33% of the sample (n = 107). For highest education level reached, 11% (n = 34) reported high school, 25% (n = 81) reported some college, 37% (n = 119) reported being a college graduate, and 28% (n = 91) reported an advanced degree. Participants on active antidepressants accounted for 34% of the group (n = 110). The mean depression score as measured by the Patient Health Questionnaire-9 (PHQ-9, Kroenke et al., 2001) was 16.8 (SD = 4.7), and the mean HAM-D score was 22.9 (SD = 4.6). Of the 325 participants included in the study, 294 completed the Week 18 end-of-treatment assessments. In addition, 28 completed fewer than five sessions and 59 completed at least 5 but fewer than 18 sessions.

2.2. Materials

2.2.1. Cognitive and Behavioral Response to Stress Scale (CB-RSS)

The Cognitive and Behavioral Response to Stress Scale was developed to measure possible cognitive and behavioral responses to stressful or upsetting situations. Two clinical psychologists generated 17 items based on the skills taught in CBT (Beck, 1995) as delivered in previous clinical trials (Mohr et al., 2001; Mohr et al., 2005; Mohr et al., 2000), including cognitive restructuring and behavioral activation, as well as other commonly used skills such as seeking social support, relaxation, and problem solving. The items were constructed to be understandable to people who have not received CBT, as well as those who have, to support a study participant’s consistent interpretation of the items both prior to treatment and after receiving CBT. These items were then reviewed for content validity and wording by two separate psychologists as well as four lay persons, who made minor modifications to the wording of the items for understandability and added two items targeting maladaptive strategies that would be expected to decrease during treatment. These modifications brought the total number of items to 19. The group of four psychologists and four lay persons all reviewed and commented on the items individually, and then met as a group to arrive at a final consensus. For each skill listed on this scale, individuals are asked to evaluate frequency and perceived usefulness. As an example, Item 1 read: During the past month, in a stressful or upsetting situation: (a) How often did you take a moment to figure out what you were feeling? 0 = Never, 1 = Rarely, 2 = Occasionally, 3 = Sometimes, 4 = Often, 5 = Very often, 6 = Always; (b) How helpful was this in making you feel better? 0 = Not at all helpful, 1 = Slightly helpful, 2 = Somewhat helpful, 3 = Moderately helpful, 4 = Fairly helpful, 5 = Very helpful, 6 = Extremely helpful, N/A = Didn’t do this last month. Participants were instructed to mark N/A for (b) if they responded with Never for (a). The full list of items is displayed in Table 1.

Table 1.

Rotated factor items for exploratory factor analysis at baseline.

Frequency Usefulness
Factor 1 Factor 2 Factor 1 Factor 2
1. Take a moment to figure out what you were feeling 0.86 0.76
2. Try to notice what you were thinking when you got upset 0.83 0.96
3. Take a moment to figure out how your thoughts made youfeel when you got upset 0.82 0.87
4. Take a moment to question your interpretation of what was happening when you got upset 0.71 0.61
5. Plan (and do) activities you knew you would enjoy 0.46 0.44
6. Take a moment to notice things that made you feel good or grateful 0.80
7. Talk with someone (friend, family, coworker, etc.) 0.50 0.43
8. Hang out with friends or people in your community (i.e., people at work, church or place of worship,social clubs, volunteer organizations, etc.) 0.90 0.44
9. Avoid other people −0.44
10. Calmly explain why you were upset or what you were upset about
11. Really try to listen to the person with whom you had the disagreement
12. Give them a piece of your mind
13. Use a relaxation technique (i.e., meditation, muscle relaxation, self hypnosis, deep breathing, listening to a relaxation tape or CD, etc.)
14. Have a drink or use something else to calm down (prescription medications, pot, etc.)
15. Sleep or watch TV
16. Work on projects around the house (i.e., gardened, painted, fixed/repaired things, cleaned, cooked, etc.) 0.66
17. Try to distract yourself with something else (i.e., read a book, watch TV, call a friend, go out, etc.)
18. Plan for the worst case scenario
19. Try to come up with several practical solutions
Open in a new tab

Note: Values lower than .30 are suppressed

2.2.2. Patient Health Questionnaire-9 (PHQ-9)

The PHQ-9 is a 9-item depression symptom measure that asks participants to rate their frequency of DSM symptoms over the past 2 weeks (Kroenke et al., 2001). In this study, Cronbach’s alpha was .78 at baseline and .90 at end of treatment.

2.2.3. Hamilton Rating Scale for Depression (HAM-D)

The HAM-D is a 17-item semi-structured interview-based measure of depression symptom severity (Hamilton, 1960). Compared to the PHQ-9, the HAM-D has more questions related to anxiety and somatic complaints and focuses more on symptom severity. Bachelor’s level clinical evaluators who had received training and supervision by a licensed PhD-level psychologist conducted all clinical interviews for this study. The mean interclass correlation of interviewer ratings was .96 (Mohr et al., 2012).

2.2.4. Positive and Negative Affect Schedule (PANAS-PA)

The PANAS-PA is a 10-item measure of positive affective states (Watson et al., 1988). Although the full scale contains independent factors of positive and negative affect, we used the positive affect items only as negative affect tends to correspond highly to measures of depressive symptoms (Crawford and Henry, 2004), whereas positive affect often has unique predictive power (e.g., Fredrickson and Joiner, 2002). In this study, Cronbach’s alpha was .87 at baseline and .95 at end of treatment.

2.2.5. Perceived Stress Scale (PSS)

The PSS is a 10-item scale measuring nonspecific appraised stress (Cohen et al., 1983). In this study, the PSS had a Cronbach’s alpha of .84 at baseline and .93 at end of treatment.

2.2.6. Coping Self-Efficacy Scale (CSE)

The CSE is a 26-item measure assessing participants’ confidence in their ability to perform various coping behaviors (Chesney et al., 2006). In this study, Cronbach’s alpha was .95 at baseline and .98 at end of treatment.

2.3. Procedures

The CB-RSS was administered as part of the battery of assessments that participants in the RCT received at baseline and end of treatment (18 weeks). We used the title “Response to Stress Scale” to mask the purpose of the assessment and to reduce the possibility that participants would endorse items more positively due to the expectation that these were skills they should have learned during the course of treatment.

2.4. Analytic Strategy

The primary goals of this study were to (a) examine the latent structure of the CB-RSS, and (b) examine the predictive validity of the emergent factors. In order to identify the underlying dimensions present in the measure, we conducted common factor analysis on the measure at baseline and end of treatment. In line with recommendations for cross-validation of solutions (Floyd and Widaman, 1995), we randomly split our sample into two groups (an initial derivation sample and a reserved cross-validation sample). General guidelines for sample size indicate that 5–10 participants per item is sufficient (Floyd and Widaman, 1995), thus with 19 items on the initial CB-RSS, we required approximately 95 to 190 participants per sample. The split sample resulted in 163 participants in the initial sample and 162 participants in the reserved sample. The initial sample is used for establishing the factor structure using exploratory factor analysis (EFA) and the reserved sample is used for confirming that factor structure using confirmatory factor analysis. This is viewed as an acceptable practice when doing analysis of an item-level factor analysis of a single measure when the number of items is small. We sought a factor solution with the EFA that would meet the following criteria: (a) satisfy Cattell’s (1966) scree test, (b) retain at least three items with salient loadings on each factor ( ≥ .30), (c) yield high internal consistency for unit-weight salient items (≥ .70), and (d) achieve simple structure with a maximum number of items retained. In order to produce such a solution we used a promax rotation, as it is an oblique solution that allows for correlations between factors produced. Confirmatory factor analysis conducted on the reserved sample sought to confirm this solution. Items that did not produce salient loadings on the identified factors were dropped from subsequent analysis. Separate factor analyses were conducted for the frequency and usefulness items to ensure that any correlations between the different ways of responding to similar stems would not bias the results. After identifying items that contributed to a consistent underlying latent structure, we returned to the full sample to (a) verify that the factor structure was consistent as end of treatment and (b) examine the validity of the CB-RSS by examining the relationship between change in CB-RSS factors and other outcome variables. We used an EFA for the end of treatment assessment of the CB-RSS. An EFA was used because we used only the items identified to load on the factors from the baseline assessment and because EFA is a more powerful technique for identifying patterns using a modified set of items.

Convergent and divergent validity was explored using hierarchical linear regression analyses with change scores from baseline to end of treatment of the PHQ-9, HAM-D, PSS, CSE, and PANAS serving as dependent variables. A residualized change score approach was used to correct for regression to the mean. Baseline scores on the CB-RSS were entered into the first block to predict change in dependent measures. In the second block, end of treatment scores on the CB-RSS were entered. Due to the possibility of multicollinearity between measures, we compared the change in R2 of these models to determine relative predictive power.

3. Results

3.1. Exploratory factor analysis: Baseline

We conducted a principal axis factor analysis for the CB-RSS at baseline using the initial sample of 163 participants. The EFA was conducted on both the frequency and usefulness items to explore whether the factor structure held regardless of how each skill was assessed. A two-factor oblique solution with promax rotation met all the stated criteria including the scree test and each factor producing at least three salient loadings. Several items failed to produce significant loadings. Thus, the accepted solution produced two factors consisting of four and five items each. Table 1 presents the rotated factor loadings for each set of items. The first factor contained four items related to cognitive skills, which were consistent across the frequency and usefulness items. This factor had an eigenvalue of 4.78 and accounted for 25.25% of the total variance in the frequency items and an eigenvalue of 6.52 and accounted for 34.34% of the variance in usefulness items. This cognitive factor demonstrated an internal consistency of .88 for frequency items and .88 for usefulness items. The second factor contained four items when using the frequency items and five items when using the usefulness items. This factor had an eigenvalue of 2.01 and accounted for 10.59% of the variance in the frequency items and an eigenvalue of 1.70 and accounted for 8.94% of the variance in the usefulness items. Items on this factor corresponded to behavioral skills. We opted to select the five items produced by the EFA on the usefulness items. Part of this was due the wording of Item 9, which asked During the past month, when you were feeling stressed or down: (a) How often did you avoid other people? and (b) How helpful was this in making you feel better? This item was dissimilar from other items as it represented a maladaptive strategy, and we believe that the wording might have confused respondents. Item 16 was retained, as it is an example of a goal-directed, adaptive behavioral strategy, and was present in the factor for the usefulness items only. For subsequent analyses, we used these five items to construct a subscale for both the frequency and usefulness items. This behavioral factor demonstrated an internal consistency of .64 for frequency items and .79 for usefulness items.

3.2 Confirmatory factor analysis: Baseline

After conducting the initial EFA, we attempted to replicate the factor structure of the two-factor oblique solution in the reserved sample of 162 participants. We first did so with the frequency items. As there is no consensus as to which fit index provides the best estimation of fit, researchers have recommended that multiple fit indices be reported for a single model (Hu and Bentler, 1998). These indices should include both absolute fit indices, which are derived from the discrepancy between the observed and implied covariance matrices, and relative or incremental fit indices, which compare the chi-square for the tested model to a null or independence model in which all measures are uncorrelated. Thus for each model, we followed the guidelines published by Hu and Bentler (1998) and looked at several measures of fit including the chi-squared statistic, Standardized Root Mean Squared Residual (SRMR), the Root Mean Square Error of Approximation (RMSEA), and the index of relative fit or the Comparative Fit Index (CFI).

For the frequency responses, the fit indices were: χ2(26) = 81.1, p < .001, SRMR = .08, RMSEA = .11, CFI = .89. These fit indices suggest poor model fit, as only the SRMR indicated good model fit. We repeated this analysis for the usefulness responses. For these responses, the fit indicators were as follows: χ2(26) = 61.7, p < .001, SRMR = .05, RMSEA = .09, CFI = .94. For these responses, 3 of the 4 fit indices suggested reasonable model fit. Therefore, the factor structure appears to replicate for the usefulness response, but not the frequency responses.

3.3. Exploratory factor analysis: End of treatment

Using the reduced 9-item CB-RSS identified through the factor analysis of the baseline items, we then returned to the full sample (N = 325) for an analysis of end of treatment administration of the scale. We conducted an EFA and employed the same criteria used for the baseline measure. Again, a two-factor oblique solution with a promax rotation met all stated criteria. Table 2 displays the rotated factor loadings for these items. The cognitive factor demonstrated an internal consistency of .94 for frequency responses and .95 for usefulness responses. The behavioral factor demonstrated an internal consistency of .77 for frequency responses and .83 for usefulness responses. Taken together, these analyses support the use of the 9-item CB-RSS (four cognitive items, five behavioral items). We used this version for subsequent analysis. At baseline, the cognitive factor had a mean of 8.90 (SD = 5.83) for frequency and 9.38 (SD = 6.39) for usefulness; means for the behavioral factor were 7.33 (SD = 3.92) and 8.67 (SD = 4.80), respectively. At end of treatment, the cognitive factor had a mean of 13.59 (SD = 5.77) for frequency and 17.85 (SD = 7.82) for usefulness; means for the behavioral factor were 11.73 (SD = 4.70) and 12.91 (SD = 5.30), respectively.

Table 2.

Rotated factor items for exploratory factor analysis at end of treatment.

Frequency Usefulness
Factor 1 Factor 2 Factor 1 Factor 2
1. Take a moment to figure out what you were feeling 0.83 0.87
2. Try to notice what you were thinking when you got upset 0.89 0.95
3. Take a moment to figure out how your thoughts made youfeel when you got upset 0.97 0.96
4. Take a moment to question your interpretation of what was happening when you got upset 0.87 0.84
5. Plan (and do) activities you knew you would enjoy 0.62 0.75
6. Take a moment to notice things that made you feel good or grateful 0.51 0.68
7. Talk with someone (friend, family, coworker, etc.) 0.58 0.73
8. Hang out with friends or people in your community (i.e., people at work, church or place of worship,social clubs, volunteer organizations, etc.) 0.84 0.72
16. Work on projects around the house (i.e., gardened, painted, fixed/repaired things, cleaned, cooked, etc.) 0.39 0.44
Open in a new tab

Note: Values lower than .30 are suppressed

3.4. Convergent and discriminant validation and comparison of frequency and usefulness

We were interested in the validity of the 9-item CB-RSS and its relationship to established outcome measures (the PHQ-9, HAM-D, PANAS, PSS and CSE) used in the clinical trial. For these analyses, we again used the complete sample. Hierarchical linear regression analyses were run for each subscale as well as a model with both subscales as independent predictors. Residualized change scores from baseline to end of treatment of the established outcome measures were entered as dependent variables in separate models. Table 3 displays the R2 values for these analyses. Across all measures, models including end of treatment scores are significantly better at predicting change scores than baseline measures. End of treatment CB-RSS scores were related to each of the outcome measure change scores at the p < .001 level.

Table 3.

R2 differences between baseline and end of treatment for outcome measure change scores.

Frequency Usefulness
Cognitive Behavioral Both Cognitive Behavioral Both
Baseline R2 EOT Δ R2 Baseline R2 EOT ΔR2 Baseline R2 EOT ΔR2 Baseline R2 EOT ΔR2 Baseline R2 EOT ΔR2 Baseline R2 EOT ΔR2
PHQ-9 0.000 0.15** 0.004 0.30** 0.000 0.31** 0.001 0.20** 0.000 0.20** 0.001 0.25**
HAM-D 0.000 0.15** 0.000 0.31** 0.001 0.33** 0.000 0.21** 0.000 0.21** 0.000 0.27**
PANAS 0.004 0.20** 0.015 0.43** 0.014 0.45** 0.037 0.27** 0.012 0.32** 0.036 0.37**
PSS 0.000 0.06** 0.000 0.18** 0.000 0.18** 0.000 0.10** 0.005 0.10** 0.007 0.13**
CSE 0.000 0.37** 0.000 0.47** 0.000 0.56** 0.005 0.43** 0.000 0.40** 0.002 0.53**
Open in a new tab
*

p< .01.

**

p< .001.

EOT = End of treatment. PHQ-9 = Patient Health Questionnaire-9. HAM-D = Hamilton Rating Scale for Depression. PANAS = Positive and Negative Affect Schedule-Positive Affect Subscale.PSS = Perceived Stress Scale. CSE = Coping Self-Efficacy Scale.

To examine the role of dose of treatment on CB-RSS scores, we ran correlations between the number of sessions completed and the residualized change scores on the CB-RSS from baseline to end of treatment. The total number of session significantly related to changes in: frequency of cognitive skills (r = .246, p < .001), usefulness of cognitive skills (r = .302, p < .001), frequency of behavioral skills (r = .147, p = .014) and usefulness of behavioral skills (r = .203, p = 001).

Table 4 displays the correlations between the CB-RSS factors at baseline and end of treatment. Table 5 displays the correlations between the CB-RSS and clinical outcome measures at baseline and end of treatment. Not surprisingly, baseline CB-RSS is more strongly related to baseline levels on the clinical outcome measures, and end of treatment CB-RSS is more strongly related to end of treatments levels on the clinical outcome measures. However, it is worth noting that the magnitude of the correlations are larger at end of treatment, with the absolute value of correlations ranging from r = .16 to r = .70, than at baseline, with the absolute value of correlations ranging from r = .03 to r = .49.

Table 4.

Correlation table of baseline and end of treatment CBT-SAS factors.

Baseline End of Treatment
Cognitive Behavioral Cognitive Behavioral
Frequency Usefulness Frequency Usefulness Frequency Usefulness Frequency Usefulness
Baseline Cognitive Frequency 1
Usefulness 0.69** 1
Behavioral Frequency 0.29** 0.33** 1
Usefulness 0.25** 0.45** 0.69** 1
End of Treatment Cognitive Frequency 0.33** 0.29** 0.07 0.17* 1
Usefulness 0.18* 0.34** 0.08 0.20** 0.84** 1
Behavioral Frequency 0.15* 0.25** 0.43** 0.35** 0.47** 0.49** 1
Usefulness 0.11 0.28** 0.34** 0.42** 0.50** 0.58** 0.86** 1
Open in a new tab
*

p< .01.

**

p< .001.

Table 5.

Correlation table of baseline CBT-SAS factors and outcome measures at baseline and end of treatment.

Baseline EOT
PHQ-9 HAM-D PANAS PSS CSE PHQ-9 HAM-D PANAS PSS CSE
Baseline
Frequency Cognitive −0.03 −0.05 0.21** −0.11 0.35 −0.03 −0.03 0.14* 0.00 0.17**
Behavioral −0.25** −0.23** 0.26** −0.15* 0.41** −0.14* −0.13* 0.19** −0.05 0.23**
Usefulness Cognitive −0.18** −0.15** 0.29** −0.19** 0.49** −0.10 −0.06 0.26** −0.09 0.32**
Behavioral −0.19** −0.16** 0.32** −0.06 0.45** −0.07 −0.04 0.19** 0.07 0.26**
EOT
Frequency Cognitive 0.01 0.05 0.08 0.11** 0.21** −0.35** −0.35** 0.45** −0.16** 0.61**
Behavioral −0.20** −0.14* 0.19** −0.02 0.30** −0.54** −0.54** 0.66** −0.39** 0.70**
Usefulness Cognitive −0.11 −0.04 0.10 0.01 0.24** −0.44** −0.44** 0.53** −0.26** 0.68**
Behavioral −0.12 −0.08 0.14* 0.04 0.24** −0.42** −0.43** 0.58** −0.26** 0.63**
Open in a new tab
*

p< .01.

**

p< .001.

EOT = End of treatment. PHQ-9 = Patient Health Questionnaire-9. HAM-D = Hamilton Rating Scale for Depression. PANAS = Positive and Negative Affect Schedule-Positive Affect Subscale.PSS = Perceived Stress Scale. CSE = Coping Self-Efficacy Scale.

4. Discussion

This paper describes the validation of a novel assessment of patients' use of cognitive and behavioral skills, which relate to concepts that are often taught in CBT. The items that were retained in the measure reflect cognitive and behavioral skills, while those that are less consistently used, such as seeking social support, relaxation, and problem-solving, did not load onto any factor and were dropped. The scale demonstrates good reliability and validity. The scale addresses several challenges in the assessment of CBT skill acquisition. First, many scales use language that is specific to CBT, whereas the items for the CB-RSS were vetted by both professionals and laypersons. As such, the CB-RSS might be better able to assess differences in people's use of skills prior to being exposed to therapy, and changes found in this measure might be more meaningful. Indeed, end of treatment scores on the CB-RSS were strongly related to changes in outcome measures. Second, this scale is brief (9 items) and relies on self-report, providing a time-effective and cost-effective method of evaluating the use of cognitive and behavioral skills. Third, the CB-RSS assesses both skill use and usefulness. The addition of the assessment of usefulness allows for an understanding of the degree to which people find using different skills to be adaptive in managing their moods. Thus, this scale offers a robust alternative to more specialized scales, while assessing the patients' experiences of skill use, which is absent from existing measures.

4.1 Frequency vs. usefulness

Both frequency and usefulness of skill use were significantly related to the established outcome measures used in the clinical trial. Overall, frequency of skill use appeared to be a somewhat stronger predictor of outcome than usefulness of skills. However, this relationship was not consistent across the cognitive and behavioral factors. Instead, for cognitive coping skills, usefulness of skills appeared to be a slightly stronger predictor than frequency. These small differences were not evaluated for statistical difference and will be the focus of further research on this measure. If this pattern of results is indeed not due to chance, there are several plausible explanations. For example, it may be easier for people to quantify use of behavioral strategies. That is, behavioral skills might be more concrete (e.g., talking with a friend, doing a positive activity), whereas cognitive skills tend to be more abstract. As such, the use of cognitive skills might be harder to track through quantity but easier to think of in terms of usefulness. Another explanation is that behavioral strategies more reliably lead to improvements (and thus frequency is more important than usefulness), but that cognitive strategies are more difficult to use effectively. In clinical practice, it is sometimes proposed that behavioral activation is a more straightforward, focused approach, and may be more useful to use when patients are more severely depressed and may have difficulty learning abstract, cognitive skills (Coffman et al., 2007). In light of this, it would seem to be more valuable to understand how people are applying cognitive strategies (e.g., usefulness) and if people are applying behavioral strategies (e.g., frequency). Nevertheless, the current investigation suggests that evaluating both frequency and usefulness is warranted.

4.2 Baseline and end of treatment

In the current analysis, end of treatment scores were better predictors of change than baseline scores, which provided little predictive power. This finding is consistent with previous work showing end of treatment skill use as a stronger predictor of depression symptoms than baseline scores (Strunk et al., 2014). It is worth noting, however, that those with higher levels of baseline cognitive and behavioral skills showed greater increases in coping self-efficacy. This is consistent with previous work that has shown that higher baseline cognitive coping skills were a predictor of depression improvement (Gibbons et al., 2015). Overall, these findings would suggest that while baseline cognitive and behavioral skills may impact some mediators of improvement, they do not influence the amount of benefit accrued during therapy. Treatment outcome is more of a function of where one ends up in terms of skills than where one starts.

4.3 Limitations

Although the CB-RSS provides some improvements compared to other measures used to assess skills typically acquired through cognitive-behavioral therapy, this study and the use of this measure have limitations as well. One limitation is that the changes in these skills may result from symptom change rather than leading to it. For example, patients might participate in more pleasurable events because they experienced a reduction in symptoms as opposed to increased pleasurable events leading to reductions in depression. Future analyses could address this by including additional time points, allowing for mediator analysis. This would help inform researchers as to whether frequency and use change at different times during treatment, and how skill use relates temporally to clinical symptom changes. Another limitation is that this investigation was conducted in the context of a clinical trial for depression. Although this increases the likelihood that skills will change over time, as these are the focus of treatment, it means that all participants are those who sought treatment and decided to participate in a research study. Levels of these skills may differ in the wider population, and additional studies could be conducted comparing clinical and non-clinical samples to better understand differences in these skills among these populations (e.g. Strunk et al., 2014). Also, if these skills are important to the development and maintenance of depression, it might be the case that the range on this measure would be somewhat restricted, impacting its ability to predict outcomes. In terms of the analytic strategy, the initial factor structure drew from the usefulness items as opposed to the frequency items, which might explain why usefulness items were more predictive of outcomes than frequency. Nevertheless, issues with some of the items that loaded on factors when frequency was used (e.g., the reverse coding inherent in “avoid other people”) supported this decision. Further, while the range of cognitive-behavioral skills assessed on this measure was fairly broad, it did not include all possible skills. For example, one item (#16) measures mastery activities around the house, but does not include mastery activities conducted outside of the home. New items could be added to address other important concepts with more specific examples. Lastly, the CB-RSS is a self-report measure assessing the patient’s perspective. It might be useful to include other observers’ ratings of one’s skills (such as family members or clinicians) to determine how these perspectives overlap.

5. Conclusion

Given that CBT focuses on teaching patients skills to use in their lives, this process should be better evaluated in the context of clinical trials. Use of cognitive and behavioral skills might be a more proximal outcome of change, and research should explore how learning these skills unfold over the course of treatment. Understanding the mechanisms of change underlying psychosocial interventions is a long-standing goal of psychological research and serves to significantly advance our understanding of disorders and the processes of change that occur during the course of treatment. The use of this measure could shed light on subgroups of patients that have been historically difficult to categorize, such as extreme non-responders to cognitive therapy (Koenig et al., 2014). Furthermore, in clinical practice, understanding whether patients are effectively applying skills might give clinicians more insight into specific points of intervention rather than simply whether or not symptoms are changing. Given that the cognitive and behavioral skills taught during the treatment of depression is not unique to that disorder, future work could analyze whether this scale is useful in other contexts. Also, additional work could explore differences in skills in the broader population. As it is, the results of this study support the initial development of the CB-RSS, and it appears to be useful at predicting outcomes in a trial of CBT for depression. It was designed to assess skills that are used in the general population and taught in treatment and addresses both cognitive and behavioral change. Overall, the assessment of skills and change in these skills during treatment was a useful predictor of clinical outcomes. The CB-RSS helped explore these relationships, and its reliability, validity, and utility suggest use for future studies.

Highlights.

  • We created and validated a novel self-report measure for cognitive behavioral skill use and perceived usefulness.

  • Changes in cognitive and behavioral skills over the course of cognitive behavior therapy were related to changes in depressive symptom severity.

  • Results of this study support the initial development of the Cognitive Behavior Response to Stress Scale, and it appears to be useful at predicting outcomes in a trial of CBT for depression.

Acknowledgments

This study was funded by research grant NIMH R01 MH059708 (PI: Mohr), Dr. Schueller was supported by research grant NIMH K08 MH102336 (PI: Schueller).

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Adler AD, Conklin LR, Strunk DR. Quality of coping skills predicts depressive symptom reactivity over repeated stressors. J Clin Psychol. 2013;69:1228–1238. doi: 10.1002/jclp.21993. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. American Psychiatric Association. Practice guidelines for the treatment of patients with major depressive disorder. 3. American Psychiatric Association; Washington, DC: 2010. [Google Scholar]
  3. Barber JP, DeRubeis RJ. On second thought: Where the action is in cognitive therapy for depression. Cognitive Ther Res. 1989;13:441–457. [Google Scholar]
  4. Barber JP, DeRubeis RJ. The ways of responding: A scale to assess compensatory skills taught in cognitive therapy. Behav Assess. 1992;14:93–115. [Google Scholar]
  5. Beck JS. Cognitive therapy: Basics and beyond. The Guilford Press; New York: 1995. [Google Scholar]
  6. Brown GK, Thase ME, Vittengl JR, Borman PD, Clark LA, Jarrett RB. Assessing cognitive therapy skills comprehension, acquisition, and use by means of an independent observer version of the Skills of Cognitive Therapy (SoCT-IO) Psychol Assessment. 2015 Sep 7; doi: 10.1037/pas0000080. Epub ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Cattell RB. The scree test for the number of factors. Multivar Behav Res. 1966;1:245–276. doi: 10.1207/s15327906mbr0102_10. [DOI] [PubMed] [Google Scholar]
  8. Chesney MA, Neilands TB, Chambers DB, Taylor JM, Folkman S. A validity and reliability study of the coping self-efficacy scale. Brit J Health Psych. 2006;11:421–437. doi: 10.1348/135910705X53155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Coffman SJ, Martell CR, Dimidjian S, Gallop R, Hollon SD. Extreme nonresponse in cognitive therapy: Can behavioral activation succeed where cognitive therapy fails? J Consult Clin Psych. 2007;75:531. doi: 10.1037/0022-006X.75.4.531. [DOI] [PubMed] [Google Scholar]
  10. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983;24:386–396. [PubMed] [Google Scholar]
  11. Crawford JR, Henry JD. The Positive and Negative Affect Schedule (PANAS): Construct validity, measurement properties and normative data in a large non-clinical sample. Brit J Clin Psych. 2004;43:245–265. doi: 10.1348/0144665031752934. [DOI] [PubMed] [Google Scholar]
  12. Floyd FJ, Widaman KF. Factor analysis in the development and refinement of clinical assessment instruments. Psychol Assessment. 1995;7:286–299. [Google Scholar]
  13. Fredrickson BL, Joiner T. Positive emotions trigger upward spirals toward emotional well-being. Psychol Sci. 2002;13:172–175. doi: 10.1111/1467-9280.00431. [DOI] [PubMed] [Google Scholar]
  14. Gibbons MBC, Thompson SM, Mack RA, Lee JK, Crits-Christoph P. The relation of baseline skills to psychotherapy outcome across diverse psychotherapies. J Clin Psychol. 2015;71:491–499. doi: 10.1002/jclp.22165. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry. 1960;23:56–62. doi: 10.1136/jnnp.23.1.56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Hollon SD, Dimidjian S. Cognitive and behavioral treatment of depression. In: Gotlib IH, Hammen CL, editors. Handbook of Depression. The Guilford Press; New York: 2014. pp. 513–531. [Google Scholar]
  17. Hu L, Bentler PM. Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychol Methods. 1998;3:424–453. [Google Scholar]
  18. Jacob KL, Christopher MS, Neuhaus EC. Development and validation of the cognitive-behavioral therapy skills questionnaire. Behav Modif. 2011;35:595–618. doi: 10.1177/0145445511419254. [DOI] [PubMed] [Google Scholar]
  19. Jarrett RB, Vittengl JR, Clark LA, Thase ME. Skills of cognitive therapy: A new measure of patients' comprehension and use. Psychol Assessment. 2011;23:578–586. doi: 10.1037/a0022485. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Kanter JW, Mulick PS, Busch AM, Berlin KS, Martell CR. The Behavioral Activation for Depression Scale (BADS): Psychometric properties and factor structure. J Psychopathol Behav. 2007;29:191–202. [Google Scholar]
  21. Koenig AM, Jarrett RB, Gallop R, Barrett MS, Thase ME. Extreme nonresponse to acute phase cognitive therapy for depression: An attempt to replicate and extend. Behav Ther. 2014;45:300–313. doi: 10.1016/j.beth.2013.12.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity measure. J GenIntern Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Mohr DC, Boudewyn AC, Goodkin DE, Bostrom A, Epstein L. Comparative outcomes for individual cognitive-behavior therapy, supportive-expressive group psychotherapy, and sertraline for the treatment of depression in multiple sclerosis. J Consult Clin Psych. 2001;69:942–949. [PubMed] [Google Scholar]
  24. Mohr DC, Hart SL, Julian L, Catledge C, Honos-Webb L, Vella L, Tasch ET. Telephone-administered psychotherapy for depression. Arch Gen Psychiat. 2005;62:1007–1014. doi: 10.1001/archpsyc.62.9.1007. [DOI] [PubMed] [Google Scholar]
  25. Mohr DC, Ho J, Duffecy J, Reifler D, Sokol L, Burns MN, Jin L, Siddique J. Effect of telephone-administered vs face-to-face cognitive behavioral therapy on adherence to therapy and depression outcomes among primary care patients. J Am Med Assoc. 2012;307:2278–2285. doi: 10.1001/jama.2012.5588. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Mohr DC, Likosky W, Bertagnolli A, Goodkin DE, Van Der Wende J, Dwyer P, Dick LP. Telephone-administered cognitive-behavioral therapy for the treatment of depressive symptoms in multiple sclerosis. J Consult Clin Psych. 2000;68:356–361. doi: 10.1037//0022-006x.68.2.356. [DOI] [PubMed] [Google Scholar]
  27. Rees CS, McEvoy P, Nathan PR. Relationship between homework completion and outcome in cognitive behavior therapy. Cogn Behav Ther. 2005;34:242–247. doi: 10.1080/16506070510011548. [DOI] [PubMed] [Google Scholar]
  28. Park CL, Folkman S, Bostrom A. Appraisals of controllability and coping in caregivers and HIV+ men: Testing the goodness-of-fit hypothesis. J Consult Clin Psych. 69:481–488. doi: 10.1037//0022-006x.69.3.481. [DOI] [PubMed] [Google Scholar]
  29. Sprangers MA, Schwartz CE. Integrating response shift into health-related quality of life research: a theoretical model. Soc Sci Med. 1999;48:1507–15. doi: 10.1016/s0277-9536(99)00045-3. [DOI] [PubMed] [Google Scholar]
  30. Stiles-Shields C, Corden ME, Kwasney MJ, Schueller SM, Mohr DC. Predictors of outcome for telephone and face-to-face administered cognitive behavioral therapy for depression. Psychol Med, Jun. 2015;16:1–11. doi: 10.1017/S0033291715001208. Epub ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Strunk DR, Hollars SN, Adler AD, Goldstein LA, Braun JD. Assessing patients' cognitive therapy skills: Initial evaluation of the competencies of cognitive therapy scale. Cognitive Ther Res. 2014;38:559–569. doi: 10.1007/s10608-014-9617-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Vitaliano PO, DeWolfe DJ, Maiuro RD, Russo J, Katon W. Appraised changeability of a stressor as a modifier of the relationship between coping and depression: A test of the hypothesis of fit. J Pers Soc Psychol. 1990;59:582–592. doi: 10.1037//0022-3514.59.3.582. [DOI] [PubMed] [Google Scholar]
  33. Watson D, Clark LA, Tellegen A. Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol. 1988;54:1063. doi: 10.1037//0022-3514.54.6.1063. [DOI] [PubMed] [Google Scholar]
  34. Weinstock LM, Munroe MK, Miller IW. Behavioral activation for the treatment of atypical depression: a pilot open trial. Behav Modif. 2011;35:403–424. doi: 10.1177/0145445511405646. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES