Development and validation of the Cognitive Behavioural Therapy Skills Scale among college students

Abstract

Background

There are many different skill components used in cognitive behavioural therapy (CBT). However, there is currently no comprehensive way of measuring these skills in patients.

Objective

To develop a comprehensive and brief measure of five main CBT skills: self-monitoring, behavioural activation, cognitive restructuring, assertiveness training and problem-solving.

Methods

University students (N=847) who participated in a fully factorial randomised controlled trial of smartphone CBT were assessed with the CBT Skills Scale, the Patient Health Questionnaire-9 (PHQ-9), the Generalised Anxiety Disorder-7 (GAD-7) and the short form of the Japanese Big Five Scale. Structural validity was estimated with exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), and internal consistency evaluated with Cronbach’s α coefficients. Construct validity was evaluated with the correlations between each factor of the CBT Skills Scale, the PHQ-9, the GAD-7 and the Big Five Scale.

Findings

The EFA supported a five-factor solution based on the original instruments assessing each CBT skill component. The CFA showed sufficient goodness-of-fit indices for the five-factor structure. The Cronbach’s α of each factor was 0.75–0.81. Each CBT skills factor was specifically correlated to the PHQ-9, GAD-7, and the Big Five Scale.

Conclusions

The CBT Skills Scale has a stable structural validity and internal consistency with a five-factor solution and appropriate content validity concerning the relationship with depression, anxiety and personality.

Clinical implications

The CBT Skills Scale will be potential predictor and effect modifier in studying the optimisation of CBT interventions.

Trial registration

CTR-000031307.

Background

Packaged cognitive behavioural therapy (CBT) is an effective treatment for depression and its acute phase response rate has previously been estimated at 44%,¹ which is as effective as second-generation antidepressant medication. However, more individualised approaches are needed to continually improve on response rates. CBT is a multi-component treatment that employs skills focused on cognition and behaviour. Flexible treatment that selects customised individual skill components help to build optimal treatment. The components provided in a typical CBT package for depression include self-monitoring (SM), behavioural activation (BA),and cognitive restructuring (CR).² In some cases, assertiveness training (AT) is added to those packages.² For some packages, BA and problem-solving therapy (PS) are implemented alone.^{3 4} These five components are the major intervention techniques of CBT for depression. Selecting components, or combinations of components, is done in accordance with patient characteristics and relies on clinical judgement. However, the empirical base consists of little evidence guiding the component selection.

There are two possible directions for optimising CBT on an empirical basis. The first direction is to derive skills, or combination of skills, that are effective for the entire population of depressed patients. Such optimal combination is yet to be discovered.^{5 6} Towards this solution, several research groups are currently conducting fully factorial randomised controlled trials of CBT optimisation.^{7 8}

Second, individualised or precision CBT interventions tailored to individual characteristics are expected to improve the remission rates.⁹ This approach examines the interaction between individual patient characteristics and each CBT component. Potentially influential effect modifiers suggested in the previous literature include the CBT skills that each patient has at baseline. Strunk et al measured cognitive therapy skills at baseline and found that patients with this skill were less reactive to sad emotions in the course of the CBT treatment, that is, they were more likely to benefit from CBT.¹⁰ Based on these results, we hypothesise that matching specific CBT skill training with those who already have that skill would promote further acquisition of the skill and then lead to prompter recovery. In other words, a strengths-based approach of drawing on patients’ pretreatment CBT skills may improve outcomes.

We have therefore decided to develop a concise scale to comprehensively assess the CBT skills for each patient. Assessing a patient’s CBT skills before the intervention may help to predict outcomes and to select appropriate intervention components. Scales measuring each of the patient’s CBT skills, such as such as SM,¹¹ BA,¹² CR,¹³ AT¹⁴ and PS,¹⁵ have been developed separately and their psychometric properties have been validated. However, each of these scales contain 8–36 items and completing them may take up to 20–30 min. It is impossible to administer all of them, in addition to other important questionnaires, in the routine clinical care. We need a concise and comprehensive measure of multi-component skills to reveal the complex interactions between CBT skills possessed by patients beforehand and the specific interventions to be selected for optimal treatment.

Objective

This study aims to develop a psychometrically sound, comprehensive, yet brief measure of five main CBT skills based on the existing questionnaires developed specifically for each skill.

Methods

Participants

Participants were 847 junior college, undergraduate and graduate students enrolled in a college or university in Kyoto or Nagoya, Japan, and participated in the Healthy Campus Trial (HCT) between September 2018 and May 2020. The HCT is a fully factorial randomised controlled trial to optimise smartphone CBT developed for mental health promotion and depression prevention in healthy college students. Details of the clinical trial are described in the protocol paper.⁸ We recruited participants by placing posters in each university, distributing brochures and novelties during their annual health checkups, and through websites and social networking services. Of the students who filled in the screening questionnaires, those who scored 14 or lower and who currently did not receive any treatment from mental health professionals were invited to the informed consent procedure. Of these, those who had provided written consent were included in the analyses.

Measurement

Cognitive and behavioural skills

We used the following established questionnaires to integrate the five constructs of cognitive or behavioural skills. We have obtained written permission from the developers of all the original scales to use parts of their items in the CBT Skills Scale.

Self-monitoring

From the Cognitive Behavioral Self-Monitoring 17-item scale originally developed by Tsuchida and Fukushima,¹¹ we used the Cognitive Monitoring subscale for measuring SM skills. This scale contains questions such as, ‘I don’t bother to think about how my actions relate to my feelings’ and ‘Sometimes I don’t understand how my actions and feelings are related to each other’. Each item was rated on a four-point Likert scale from 0=very untrue of me to 3=very true of me, with a total score between 0 and 15. Cronbach’s α reported in a previous study of Japanese university students was 0.77, indicating sufficient internal consistency.¹¹

Cognitive restructuring

The six highest loading items from the Competencies of the Cognitive Therapy Scale developed by Strunk et al¹³ were used for measuring CR skills. The questions include, ‘When I became distressed because of a negative thought or feeling, I come up with a specific plan of action for what I could do to deal with it’, and ‘When something upset me, I paid attention to what I was thinking so I could have a more balanced view’. Items are rated from 0=not very true about me, to 3=very true about me, with a total score between 0 and 18. We confirmed the semantic equivalence of the Japanese and English versions by translation and back translation.

Behavioural activation

We used the five-item Behavioral Activation subscale of Behavioral Activation for Depression Scale-Short Form (BADS-SF) developed by Manos et al¹² for measuring BA skills. BADS-SF was translated to Japanese from the English version and validated in the Japanese population.¹⁶ The scale includes items such as, ‘I’m satisfied with the amount and type of things I’ve done’. Items are rated from 0=not very true in my mind to 3=very true in my mind. Cronbach’s α reported in a previous study of Japanese university students was 0.71, indicating sufficient internal consistency.¹⁶

Assertiveness training

We used the seven-item Self-Assertion subscale of the Adult Social Skills Scale developed by Aikawa and Fujita¹⁴ for measuring AT skills. The scale includes items such as, ‘I complain clearly when I’m made uncomfortable’. Items are rated between 0=not very true about oneself to 3=very true about oneself. Cronbach’s α reported in a previous study of Japanese university students was 0.73, indicating sufficient internal consistency.¹⁴

Problem-solving

The six highest loading items of the Approach Avoidance Style subscale of the Problem-Solving Inventory¹⁵ were used for measuring PS skills. A Japanese version of the inventory was translated and back translated. The scale includes items such as, ‘When making a decision, I weigh the consequences of each alternative and compare them against each other’. Items are rated from 0=very untrue of me to 3=very true of me, with a total score ranging from 9 to 18.

Mood and personality

Depression

The Patient Health Questionnaire-9 (PHQ-9)¹⁷ was used for measuring depression. The PHQ-9 uses 9 items of diagnostic criteria for major depressive episode (MDE) in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, and is rated from 0=not at all to 3=nearly every day. The Japanese version of the PHQ-9 has been shown to be valid and reliable in primary care patients¹⁸ and the university students.¹⁹

Anxiety

The Generalised Anxiety Disorder-7 (GAD-7)²⁰ was used for measuring anxiety. The GAD-7 is an established measure including 7 items of anxiety, worry and hypersensitiveness. The scale is rated from 0=not at all to 3=nearly every day. The Japanese version of the PHQ-9 has been shown to be valid and reliable in primary care patients with anxiety disorder or major depressive disorder.²¹

Personality

For investigating the correlation between the CBT skills and personality traits, we used the short form of the Big Five Scale of Personality Traits (Big Five Scale), which is well established in reliability and validity, and is frequently used in Japan.^{22 23} The Big Five Scale is rated from 0=untrue for me to 4=true for me on each of the five personality trait factors: neuroticism, extraversion, openness, conscientiousness and agreeableness.

Data collection

Data were collected as an assessment before participants were enrolled in the HCT and assigned to each intervention group, and all participants completed all questionnaires on the trial registration website and the smartphone CBT app on their smartphones. Of these, those participants who gave consent for the use of the data were used for the current analysis. The HCT is being conducted with the approval of the Ethics Committee of Kyoto University School of Medicine. We analysed the cross-sectional data of baseline instruments of the trial in this investigation.

Statistical analysis

Data analysis was conducted with SPSS V.25 for investigating construct validity and internal consistency. Amos V.22 (IBM SPSS Statistics, Chicago, Illinois, USA) was used for confirmatory factor analysis (CFA).

Structural validity

We first conducted an exploratory factor analysis (EFA) using maximum likelihood and Promax rotation to identify the factor structure of the five CBT Skills Scales collectively. Next, a CFA was performed based on the factor loadings of each factor to confirm the goodness-of-fit for the factor structure. χ² (CMIN), goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI), comparative fit index (CFI) and root mean square error of approximation (RMSEA) were calculated. The criterion for a good model fit was set to CMIN/df≤2, GFI≥0.95, AGFI≥0.90, CFI≥0.97 and RMSEA≤0.05. An acceptable model fit was set to CMIN/df≤3, GFI≥0.90, AGFI≥0.85, CFI≥0.95 and RMSEA≤0.08.²⁴

Reliability

Cronbach’s α coefficients were calculated to test the internal consistency reliability of each of the five CBT Skills Scales. The α value considered to be of sufficient internal consistency ranges from 0.7 to 0.8.²⁵

Construct validity

Pearson’s correlation coefficients were calculated to examine the correlations between the five CBT Skills Scales and symptoms of depression, anxiety and specific personality traits, as measured respectively with PHQ-9, GAD-7 and the Big Five Scale. We had hypothesised that symptoms of depression and anxiety would correlate negatively with the CBT skills, and that neuroticism would correlate negatively with the CBT skills while agreeableness and extraversion would correlate positively with some CBT skills especially representing assertion or problem solving.

Findings

Participant characteristics

Table 1 shows the participants’ baseline characteristics. Participants (N=847) were a mean age of 22.0 (SD=3.0, range=18–39) and mostly women (53.4%). Most participants were undergraduates (70.1%), with 21.7% in master’s programmes, 7.8% in doctoral programmes and 0.4% in junior college programmes. Means and SD of PHQ-9 and GAD-7 score were 6.4 (3.4) and 5.2 (3.3).

Table 1.

Participants’ characteristics (n=847)

Variables	Mean (SD) or numbers (%)
Age	22.0 (SD=3.0)
Sex (%)
Female	463 (53.4%)
Male	401 (46.6%)
Programme (%)
Bachelor	610 (70.1%)
Master	185 (21.7%)
Doctoral	66 (7.8%)
Junior college	3 (0.4%)
PHQ-9	6.4 (SD=3.4)
GAD-7	5.2 (SD=3.3)

GAD-7, Generalised Anxiety Disorder-7; PHQ-9, Patient Health Questionnaire-9.

Structural validity

On the CBT Skills Scale, five factors were extracted according to the EFA (table 2). The factor loadings for items of each skill were highest among the items in the original scale; 0.41–0.81 for SM, 0.33–0.74 for BA, 0.43–0.79 for CR, 0.22–0.76 for AT and 0.50–0.69 for PS. Therefore, the CFA assumed a model of the CBT Skills Scale by combining the original scales and calculated the goodness-of-fit (figure 1). The results showed that CMIN/df=2.20, GFI=0.93, AGFI=0.92, CFI=0.93 and RMSEA=0.04, with AGFI and RSMEA being good and CMIN/df, GFI and CFI being at acceptable levels.

Table 2.

Factor loadings of the Cognitive Behavioural Therapy Skills Scale from the maximum likelihood factor analysis with five Promax rotation (n=847)

Items	Factors
	BA	CR	AT	SM	PS
BA05	0.743	0.038	0.016	0.026	−0.052
BA06	−0.692	0.071	0.062	0.007	−0.109
BA02	0.652	0.049	−0.028	−0.010	−0.087
BA04	0.633	0.057	0.047	−0.035	−0.033
BA03	0.595	−0.018	0.077	0.081	−0.021
BA07	−0.542	0.059	0.051	0.044	−0.061
BA01	−0.474	0.076	0.094	0.015	−0.024
BA08	0.329	0.043	0.069	0.016	0.001
CR03	−0.042	0.788	−0.061	0.004	−0.081
CR04	−0.024	0.708	−0.066	−0.011	−0.008
CR06	−0.058	0.681	0.055	−0.008	−0.024
CR02	0.077	0.658	−0.001	−0.051	−0.005
CR05	−0.048	0.585	0.014	0.069	0.064
CR01	0.186	0.428	0.017	−0.010	0.064
AT01	0.007	−0.030	0.761	0.051	−0.050
AT05	0.012	−0.067	0.685	−0.032	−0.056
AT02	−0.021	0.011	0.659	0.083	−0.071
AT04	−0.020	−0.018	0.597	0.021	0.052
AT03	−0.025	−0.023	0.452	−0.054	0.055
AT07	−0.037	0.139	0.369	−0.074	0.106
AT06	0.188	0.039	0.217	−0.111	0.037
SM02	0.006	0.014	−0.048	0.813	0.042
SM03	0.020	0.063	−0.019	0.785	−0.039
SM01	−0.007	−0.060	0.015	0.697	−0.020
SM04	0.004	0.039	0.034	0.454	0.008
SM05	−0.030	−0.064	0.046	0.408	0.047
PS05	0.010	−0.014	0.029	−0.026	0.692
PS04	−0.065	0.079	−0.042	0.027	0.634
PS03	−0.038	0.110	0.026	−0.022	−0.596
PS06	−0.056	0.077	0.089	−0.025	0.554
PS02	−0.012	0.074	0.086	0.027	−0.498
PS01	0.093	0.075	0.061	0.095	0.496

AT, assertiveness training; BA, behavioural activation; CR, cognitive restructuring; PS, problem-solving; SM, self-monitoring.

Figure 1.

Confirmatory factor analysis of a five-factor model of the Cognitive Behavioural Therapy Skills Scale. AT, assertiveness training; BA, behavioural activation; CR, cognitive restructuring; PS, problem-solving; SM, self-monitoring.

Internal consistency reliability

The Cronbach α of each CBT Skills Scale was 0.78 for SM, 0.80 for BA, 0.81 for CR, 0.75 for AT and 0.75 for PS, all of which suggest satisfactory internal consistency reliability.

Construct validity

The correlation coefficients for each CBT skill with the five-factor personality, depression, and anxiety were calculated for construct validity and are shown in table 3. Depression symptoms were negatively correlated with BA and SM and less strongly so with CR and AT. Anxiety symptoms were negatively correlated with SM and less strongly with BA, CR and AT.

Table 3.

Pearson’s correlation between the CBT Skills, the PHQ-9, the GAD-7 and the Big Five Personality (n=847)

	SM	BA	CR	AT	PS	PHQ-9	GAD-7	Neuroticism	Extroversion	Openness	Conscientiousness	Agreeableness
CBT skills
SM	1
BA	0.17** (0.10 to 0.23)	1
CR	0.11** (0.04 to 0.17)	0.36** (0.30 to 0.42)	1
AT	0.11** (0.04 to 18)	0.24** (0.17 to 0.30)	0.26** (0.20 to 0.32)	1
PS	0.07 (0.00 to 0.14)	0.20** (0.13 to 0.26)	0.34** (0.28 to 0.40)	0.21** (0.14 to 0.27)	1
PHQ-9	−0.35** (−41 to −0.29)	−0.37** (−0.42 to −0.31)	−0.19** (−0.25 to −0.12)	−0.21** (−27 to −0.14)	−0.05 (−12 to 0.02)	1
GAD-7	−0.24** (−30 to −0.18)	−0.17** (−0.23 to −0.10)	−0.08 (−0.15 to −0.01)	−0.17** (−0.23 to −0.10)	−0.02 (−0.09 to 0.05)	0.43**  (0.37 to 0.48)	1
Big Five
Neuroticism	−0.24** (−30 to −0.18)	−0.23** (−0.29 to −0.17)	−0.18** (−24 to −11)	−0.31** (−0.37 to −0.25)	0.02 (−0.05 to 0.09)	0.33**  (0.27 to 0.39)	0.51** (0.46 to 0.56)	1
Extroversion	0.05 (−0.02 to 0.12)	0.25** (0.19 to 0.31)	0.19** (0.12 to 0.25)	0.34** (0.28 to 0.40)	0.07* (0.00 to 0.14)	−0.15**  (−0.22 to −0.08)	−0.08 (−0.15 to −0.01)	−0.20**  (−0.26 to −0.13)	1
Openness	−0.01 (−0.08 to 0.06)	0.22** (0.15 to 0.28)	0.28** (0.22 to 0.34)	0.33** (0.27 to 0.39)	0.22** (0.15 to 0.28)	−0.10**  (−0.17 to −0.03)	−0.02 (−0.09 to 0.04)	−0.18**  (−0.24 to −0.11)	0.40** (0.34 to 0.46)	1
Conscientiousness	0.05 (−0.02 to −0.12)	0.30** (0.24 to 0.36)	0.24** (0.18 to 0.30)	0.06 (−0.01 to 0.13)	0.37** (0.31 to 0.43)	−0.15**  (−0.22 to −0.08)	0.03 (−0.04 to 0.1)	0.05  (−0.02 to 0.11)	−0.06 (−0.13 to 0.01)	0.10** (0.03 to 0.17)	1
Agreeableness	0.12** (0.05 to 0.09)	0.16** (0.09 to 0.22)	0.21** (0.14 to 0.27)	−0.01 (−0.08 to 0.06)	0.20** (0.13 to 0.26)	−0.11**  (−0.18 to −0.04)	−0.23** (−0.29 to −0.17)	−0.16**  (−0.22 to −0.09)	0.06 (−0.01 to 0.13)	0.19** (0.12 to 0.25)	0.21** (0.14 to 0.27)	1

Values in parentheses show the 95% CIs.

**p<0.01, *p<0.05.

AT, assertiveness training; BA, behavioural activation; Big Five, Big Five Scale of Personality Trait; CR, cognitive restructuring; GAD-7, Generalised Anxiety Disorder-7; PHQ-9, Patient Health Questionnaire-9; PS, problem-solving; SM, self-monitoring.

With regard to the Big Five personality Traits (1) extraversion was weakly and positively correlated with AT and BA, (2) conscientiousness was weakly positively correlated with BA, CR and PS, (3) openness was weakly and positively correlated with AT, BA, CR and PS, (4) agreeableness was weakly and positively correlated with CT and PS and (5) neuroticism was weakly negatively correlated with AT, BA and SM.

Discussion

The purpose of this study was to develop a brief and comprehensive measure of the five major CBT skills and to establish the validity and reliability of a measure of said skills. To investigate this, a factor structure was validated by EFA and CFA. Then, internal consistency was verified by calculating the Cronbach’s α. Moreover, construct validity was tested by investigating the correlation between each CBT skill and the potential outcomes of depression, anxiety and personality traits.

The present findings contribute to the literature by developing a useful instrument that is necessary to determine the yet to be discovered optimal CBT components and combinations.^{5 6} The results of the EFA showed that SM, BA, CR, AT and PS could indeed be classified as distinct factors representing separate CBT skills. Additionally, the CFA confirmed that the factor structure of the CBT Skills Scale is appropriate. Furthermore, Cronbach’s α for each skill showed acceptable internal consistency. Each of the five CBT skills (SM, BA, CR, AT and PS) could be considered separately to measure specific skills in each intervention factor. These findings will allow researchers to validate the matching of skills with each intervention in clinical trials using multiple components of CBT. Such studies would then contribute to the optimisation of interventions in clinical practice according to patient characteristics.

The present findings advance existing research in individualised treatment^{9 26} by demonstrating that individual CBT skills can be differentially associated with varied potential mental health outcomes. We identified significant and differential associations between each CBT skill and potential outcomes of depression, anxiety and personality traits. In the present findings, we found that the higher the SM, BA and AT, the lower the depression symptoms. Additionally, we found that the higher the SM, the lower the anxiety. In the treatment of both depression and anxiety disorders, SM-based assessments are one of a key component of CBT and SM itself promoted symptom reduction by objectively observing the problem related emotions, cognitions and behaviours.²⁷ Because BA and AT are activities that directly antagonise depressive behaviours,²⁸ it appears that a particularly relevant association with depression was found in this cross-sectional investigation.

Concerning each skill and the personality traits measured by the Big Five Scale, specific correlations were found between each trait and each skill. Higher openness tended to be associated with higher levels of all skills except SM, which suggests that being open to new experiences makes it easier to activate a wide variety of CBT skills. Higher extroversion was associated with higher skills in BA and AT, which suggests that these skills are more likely to be honed by relationships with others. Higher conscientiousness was associated with higher levels of BA, CR and PS, which are skills commonly assigned as homework of continuous habits in CBT. This may mean that those low in conscientiousness or low in pretreatment BA, CR and PS, may struggle in treatment adherence. Those who were more agreeable tended to have higher CR and PS skills, which may indicate that they are better at the demonstrating empathy. On the other hand, neuroticism tended to have lower skills in SM, BA and AT, which may indicate that they may have more skills to develop as targets for CBT. These findings are consistent with the results of a meta-analysis that examined the relationship between resilience, which addressed in each component of CBT, and the Big Five personality traits.²⁹ Neuroticism was negatively correlated with resilience and other personality traits were positively correlated with resilience.

The optimisation of CBT components and combination their combinations is important to reduce depressive symptoms and prevent MDEs. The CBT Skills Scale developed in this study will enable researcher and potentially clinicians to predict the effect of each intervention element or combination of interventions and to explore effect modifiers. It has the potential to help quantitatively refine and individualise CBT and provide optimal interventions.

Limitations

The participants of the current study were college students below the depression threshold, which may differ from those with clinical depression. However, for those with higher levels of depression, the CBT skills and outcome associations may be even stronger. Future studies should be conducted in a variety of patient groups with differing characteristics and symptom severity. Second, the present study only looked at the association between CBT skills and cross-sectional depression and anxiety, and the association with their longitudinal changes is not clear. The association will be investigated after the trial is completed. Third, we would need broader participant characteristics including but not limited to CBT skills to adequately build a personalised prediction model for the CBT intervention.

Conclusions

The CBT Skills Scale showed stable structural validity, sufficient internal consistency and sufficient reliability. Importantly, we demonstrated that there is a five-factor model of CBT skills aligned with the five most commonly used CBT intervention components. The scale also demonstrated an interpretable relationship with depression, anxiety and the specific personality traits.

Clinical implications

The CBT Skills Scale could be used as a potential predictor and effect modifier in studying the optimisation of CBT interventions. This scale may then help with clinical decision-making by conceptualising patient strengths associated with component selection in relation to differential outcomes.

Data availability statement

No data are available. Data sharing is not applicable as the scope of data use is limited to informed consent to participants.

Ethics statements

Patient consent for publication

Not required.

Ethics approval

The Ethics Committee of Kyoto University School of Medicine have approved this study (Protocol # C1357). All participants provided written informed consent.