Abstract
Knowledge gain has been identified as necessary but not sufficient for therapist behavior change. Declarative knowledge, or factual knowledge, is thought to serve as a prerequisite for procedural knowledge, the how to knowledge system, and reflective knowledge, the skill refinement system. The study aimed to examine how a one-day workshop affected therapist cognitive behavioral therapy declarative knowledge. Participating community therapists completed a test before and after training that assessed cognitive behavioral therapy knowledge. Results suggest that the workshop significantly increased declarative knowledge. However, post-training total scores remained moderately low, with several questions answered incorrectly despite content coverage in the workshop. These findings may have important implications for structuring effective cognitive behavioral therapy training efforts and for the successful implementation of cognitive behavioral therapy in community settings.
Keywords: Cognitive Behavioral Therapy, Training, Declarative Knowledge, Assessment
Although several Empirically Supported Treatments (ESTs), including Cognitive Behavioral Therapy (CBT), for depression and anxiety disorders (Butler, Chapman, Forman, & Beck, 2006) have been identified, a well-documented gap continues to exist between scientific research and applied clinical practice (Shafran et al., 2009). There is a growing body of literature demonstrating the effectiveness of CBT in community settings (Gibbons et al., 2010; Merrill, Tolbert, & Wade, 2003; Simons et al., 2010); however, the structure of training necessary to effectively enable therapist implementation of CBT is not well understood. Several studies have evaluated CBT training efforts with promising results. Simons et al. (2010) conducted training for community practitioners that included a two-day CBT workshop followed by one year of tri-weekly group phone consultation. At six and 12 months post workshop, therapists demonstrated CBT competence scores comparable to those achieved by research therapists in randomized clinical trials. In a similar study, CBT training was held across 10 days using a combination of workshop training and case supervision. Following training, therapists demonstrated significant improvements in CBT competence (Westbrook, Sedgwick-Taylor, Bennett-Levy, Butler, & McManus, 2008). Both studies also noted superior outcomes for clients who received CBT from trained clinicians when compared to clients who received treatment as usual (TAU; Simons et al., 2010; Westbrook et al., 2008). These findings reflect a growing body of literature in which competent CBT implementation outperforms TAU in community settings (Gibbons et al., 2010; Merrill et al., 2003). All known successful models of CBT implementation have a common structure: workshop followed by consultation. In order to evaluate the utility of this common structure for promoting therapists’ competent implementation of CBT and improving community mental health care, it is first necessary to understanding the knowledge gain that can be expected as a result of workshop training.
CBT Training Techniques and Knowledge Gain
A crucial step in promoting competent CBT implementation and enhancing the use of CBT in community mental health settings is to improve therapists’ knowledge. Jensen-Doss, Cusack, & de Arellano (2008) suggested that knowledge change is a necessary precondition for therapist behavior change. According to Bennett-Levy, McManus, Westling, & Fennell (2009), knowledge gain occurs across three systems. The declarative knowledge system refers to specific CBT conceptual knowledge (i.e. knowledge of facts; e.g.the CBT model), while the procedural knowledge system involves the development and application of CBT skills in clinical practice. The reflective knowledge system, which represents ongoing refinement of declarative and procedural knowledge, allows therapists to reflect and solve problems in clinical practice and is acquired through ongoing experience (Bennett-Levy et al., 2009). Declarative knowledge must be acquired before techniques can be applied procedurally (McCall, Arnold, & Sutton, 2008), and both declarative and procedural knowledge must exist before a therapist can gain reflective skills (Bennett-Levy et al., 2009). Traditionally, declarative and procedural knowledge gains occur in the context of training and consultation, while reflective knowledge is acquired through active application of techniques during and after therapy sessions (Bennett-Levy, 2006).
Given the staged progression of knowledge development from declarative to procedural to reflective, it is important to consider the training techniques that will maximize therapist knowledge gain, especially with respect to the declarative system, to facilitate therapist competent CBT implementation. Substantive literature exists highlighting the limitations of traditional continuing education unit (CEU) workshops for achieving therapist behavior change (Forsetlund et al., 2009). However, these brief workshops remain a popular and potentially effective training structure, thus it is imperative to understand their utility for improving CBT declarative knowledge. Specific training strategies, such as lectures (declarative) and role plays (procedural), appear to have differential effects on knowledge gain across systems, which may be critical to enhancing the value of workshop training structures (Bennett-Levy et al., 2009).
Measuring CBT Knowledge Gain
To build this area of research, it is important to evaluate knowledge gain in order to gauge the effectiveness of training efforts. However, few knowledge tests have been developed to evaluate declarative knowledge gain. One existing CBT knowledge test developed by Myles & Milne (2004) has demonstrated sensitivity to change in CBT knowledge pre and post training among primary care practitioners and in other treatment settings (Maunder, Milne, & Cameron, 2008). This particular knowledge test allows evaluation of declarative knowledge; however, procedural and reflective knowledge are thought to require procedures such as behavioral rehearsal (Beidas, Cross, & Dorsey, 2014), role-plays, or reflective practice exercises (Bennett-Levy et al., 2009) for accurate evaluation. Since procedural and reflective knowledge involve practical application and refinement through active clinical practice outside of training settings (Bennett-Levy et al., 2009), it cannot be expected that a self-report knowledge measure would suffice in accurately characterizing these knowledge systems, nor should it be expected that workshop trainings would significantly enhance knowledge in these two systems.
A variety of training strategies exist to improve declarative CBT knowledge. However, previous research has largely failed to identify specific declarative CBT knowledge topics that may change as a result of workshop training. Identifying the CBT knowledge topics that do or do not change as a result of workshops may prove helpful in developing and structuring training efforts. The focus of the present study was to evaluate the utility of a one-day workshop to enhance community mental health center (CMHC) therapists’ understanding of declarative CBT knowledge topics. The primary aims of the study were three-fold: (1) to examine the effect of a one-day workshop on CMHC therapist CBT declarative knowledge; (2) to investigate the effect of therapist variables at baseline (i.e. demographics and prior CBT exposure) on CBT knowledge acquisition; and (3) to assess specific CBT knowledge topics prior to and following training. We hypothesized that a one-day workshop would lead to significant increases in CBT declarative knowledge. Limited research regarding the influence of baseline therapist variables suggests that there are few predictors (i.e. degree level and clinical setting) of EST knowledge gain (Nakamura, Higa-McMillan, Okamura, & Shimabukuro, 2011). Thus with respect to the second aim, we hypothesized that change in CBT knowledge would not significantly differ based on therapist baseline variables (i.e. age, gender, theoretical orientation). Finally, our third aim was exploratory in nature and no a priori hypotheses were specified.
Methods
Participants
Participants were CMHC therapists (N=38) representing 11 community agencies in Lane County, Oregon who volunteered to participate in a CBT workshop for treating adolescent and adult depression. The workshop was open to all therapists in the community. Therapists were compensated with CEUs for their workshop participation.
Procedures
Training
The training was funded by a public insurance company in Lane County, Oregon and was delivered by the University of Oregon Psychology Clinic Dissemination Team. CBT training was provided by a founding fellow of the Academy of Cognitive Therapy, a certified CBT Diplomate with 30 years of experience training practitioners in CBT. The workshop focused equally on four CBT modules introduced in the following order: case conceptualization, behavioral activation, thought records, and behavioral experiments. The workshop included didactics (60% of day), live (10% of day) as well as video (5% of day) demonstrations, and experiential exercises (25% of day; e.g., completing a personal thought record) to explicitly target declarative knowledge.
Data Collection Design
Prior to the workshop, therapists completed a battery of questionnaires to assess demographic variables (e.g., gender, age, ethnicity) and previous exposure to CBT. Therapists also completed a paper-and-pencil test of CBT knowledge (a modified version of the Cognitive Behavioral Therapy Knowledge Questionnaire; CBTKQ; see below). Immediately following the workshop, therapists completed the CBTKQ a second time to assess knowledge gain as a result of the one-day training. All study procedures were reviewed and approved by the University of Oregon Institutional Review Board.
Measures
Demographics and Baseline Therapist Variables
The demographic questionnaire assessed variables including gender, age, ethnicity, clinical experience, theoretical orientation, and experience with giving and receiving supervision. The questionnaire also assessed prior CBT exposure including previous training, textbook exposure, research article consumption, supervision involvement (both as a supervisee and supervisor), and formal education (i.e., a graduate level seminar).
CBTKQ
CBT knowledge was assessed through a modified version of the Cognitive Behavioral Therapy Knowledge Questionnaire (CBTKQ; Latham, Myles, & Ricketts, 2003). The CBTKQ developed by Latham and colleagues (2003) contained 26 multiple-choice questions from five conceptual topics: general CBT issues, theoretical underpinnings of behavioral approaches, theoretical underpinnings of cognitive approaches, practice of behavioral psychotherapy, and practice of cognitive therapy. The modified CBTKQ used in the present study incorporated 14 questions from the original CBTKQ with anxiety items omitted as the training focused on CBT for depression. Additional items were incorporated from a knowledge test developed by Simons and Padesky (2010). The modified CBTKQ contained a total of 26 multiple-choice questions with four response options per question, as well as one question in which participants were asked to place therapy components in the order they would typically be employed in CBT. The CBTKQ examines declarative knowledge items including definitions of negative automatic thoughts, “Downward Arrow” technique, determining the initial focus of CBT and placing therapy components in the order of application in CBT. Although several of these items (e.g. order of therapy components) refer to procedural aspects of CBT, they require the therapist to demonstrate knowledge of the factual information regarding the procedure rather than demonstrating their competency in procedure implementation. Therefore, the CBTKQ serves only to assess therapist declarative knowledge of CBT (Simons, Rozek, & Serrano, 2013).
Statistical Analyses
Baseline
Student t-tests were used to compare pre-training CBTKQ total scores across dichotomous demographic variables. General Linear Models (GLMs) were employed to compare CBTKQ scores across multi-option categorical demographic variables. Common pre-training correct and incorrect CBTKQ answers were explored (common refers to questions in which at least 75% of participants selected the correct or incorrect answer).
Post-Training
Pre-post workshop change in declarative CBT knowledge was evaluated by paired samples t-tests. Student t-tests and GLMs were used to determine whether there were significant differences in post-workshop CBTKQ scores across the demographic factors while controlling for pre-workshop scores. Change in knowledge was explored by examining changes in correct and incorrect answers to specific CBT knowledge topics post training.
Results
Participants
Participants were 38 CMHC therapists, aged 26 to 61 (M = 45.26, SD = 9.62), from 11 community agencies. Seventy-one percent were female and 86.8% were Caucasian. Therapists had a wide range of experience, with 35.3% having 10 to 20 years experience and 23.5% having 1 to 3 years. A majority of therapists had completed a Master's Degree or higher (94.7%) and 33.3% self-identified as supervisors. A minority of therapists defined themselves as Cognitive Behavioral Therapists (23.5%). Prior exposure to CBT was minimal: 76.5% had not been supervised in CBT and 67.6% had received no formal CBT education in their graduate degree program. However, 69.7% of therapists reported they had previously attended a CBT workshop.
Pre-Training
Participating therapists scored just over 50% (M = 13.55, SD = 3.63) on the CBTKQ declarative knowledge questions prior to training. No significant differences were observed on the pre-training CBTKQ with respect to baseline therapist demographics (e.g., gender, age, ethnicity, experience, theoretical orientation, and previous exposure to CBT). Subsequent analyses of the CBTKQ identified items that were answered either correctly or incorrectly by at least 75% of therapists. Six questions were answered correctly by a majority of therapists, including declarative items evaluating knowledge of the therapeutic relationship, core beliefs, the seven-column thought record, cognitive case conceptualization, activity scheduling and outcome monitoring. Four questions were answered incorrectly by a majority of therapists, including items evaluating knowledge of the initial focus of CBT, the cognitive triad, behavioral experiments, and the order of CBT techniques.
Post-Training
Paired samples t-tests revealed that overall therapist knowledge test scores on the CBTKQ significantly improved from pre (M = 13.55, SD = 3.63) to post (M = 17.26, SD = 2.97) workshop reflective of a large effect size (t (37) = -6.68, p < 0.001, d = 1.12). However, therapists only answered 66.4% of items correctly on the post training knowledge test, with a majority of therapists (more than 75%) answering only 13 of the 26 items correctly. Two questions regarding the cognitive triad and ordering of CBT techniques were answered incorrectly by a majority of therapists despite workshop training. Additionally, questions regarding the definition of cognitive therapy, initial focus of CBT, underlying assumptions, the earliest stages in treating depression, and graded task assignment were answered incorrectly by more than 50% of therapists. Student t-tests and GLMs revealed no differences in post-workshop CBTKQ scores across therapist demographic factors when controlling for pre-workshop test scores.
Discussion
Previous research has shown that therapist knowledge acquisition may serve as a moderator for behavior change (Davis et al., 1999; Jensen-Doss et al., 2008). The present study sought to examine the impact of a typical CEU-style workshop on CMHC therapist CBT declarative knowledge, as workshops may be an optimal medium for enhancing declarative knowledge on which therapists can build procedural and reflective knowledge through ongoing consultation. Consistent with the hypothesis and the literature, a one-day workshop led to significant increases in therapist overall declarative knowledge of CBT. With respect to the second aim, and also consistent with previous research, there were no differences in knowledge gain based on baseline therapist factors. With respect to the third aim, persistent CBT declarative knowledge errors emerged.
Implications for CBT Training and Knowledge Gain
This study replicates previous research showing that a one-day CBT workshop results in CBT knowledge gain among CMHC therapists (Maunder et al., 2008; Myles & Milne, 2004; Sholomskas et al., 2005). Although the majority of therapists reported prior exposure to CBT, average declarative CBT knowledge scores were quite low at baseline. A potential reason for this poor initial performance despite prior exposure is the fact that CBT is an incredibly broad area of study, therefore the focus of therapists’ previous training may not have aligned with the content of the administered CBT test.
The majority of therapists (greater than 75%) answered several declarative knowledge items correctly that were focused on definition of terms and use of CBT specific interventions prior to the workshop. Simultaneously, at baseline, therapists demonstrated incorrect declarative knowledge about CBT, primarily with respect to the focus and ordering of CBT interventions throughout treatment. These items are associated with procedural (i.e. how to) aspects of CBT; therefore they may require unique or more active training strategies to facilitate knowledge gains in workshop training. Overall, identifying and addressing these incorrect beliefs early in a CBT implementation effort may be particularly important to enhance the effects of workshop structured training and subsequent consultation.
Therapists’ CBT declarative knowledge was significantly and positively influenced by the workshop. There was a significant increase in the number of CBT declarative knowledge questions that the therapists answered correctly. Bennett-Levy and colleagues (2009) and McCall and colleagues (2008) suggest that these knowledge gains are necessary to enable the therapist to implement (procedural knowledge) and problem solve (reflective knowledge) CBT techniques in applied clinical practice. Thus increases in the declarative knowledge system suggest some progress towards therapists’ ability to implement CBT in practice. Additionally, this knowledge gain obtained may enhance clinician attitudes about CBT training and encourage them to seek out additional opportunities to develop procedural and reflective knowledge (Borntrager, Chorpita, Higa-McMillan, & Weisz, 2009)
Despite increases in declarative knowledge as a result of workshop training, participants only achieved an average of 66% correct on all declarative items. These results suggest that the training strategies employed in the one-day workshop may not have been sufficient to promote an understanding of all covered CBT topics. Knowledge may be difficult to shift through during a one-day workshop training, as large amounts of information may max out working memory capacity (de Jong, 2010). CBT knowledge gain and implementation via a one-day workshop training approach may be restricted by both the quality of the training and the amount of information that can be grasped by therapists in a limited time frame. Training quality likely varies across instructors given differences in fidelity to training strategies utilized (e.g. didactic teaching, role-plays; Cross et al., 2014). It may be the case that longer workshops including didactic and experiential exercises (Gleacher et al., 2011; Westbrook et al., 2008) are necessary to reduce the cognitive load associated with CBT training and maximize knowledge gain potential for all participating therapists.
Limitations and Future Directions
While self-report knowledge tests, such as the one employed in this study, may be useful for assessing declarative knowledge gain, they fail to assess changes in procedural and reflective knowledge. As demonstrated by Bennett-Levy and colleagues (2009), role-playing, reflective practice, and experiential techniques may be most useful for both enhancing and evaluating procedural and reflective knowledge. Although the CBTKQ had questions assessing declarative knowledge of the how to aspects of CBT, future research should explore how role-play or observational assessment techniques may be efficiently employed to evaluate gains in procedural and reflective knowledge. Behavioral rehearsal assessment techniques that allow the clinician to demonstrate their knowledge through simulated interaction may also be effective and efficient for assessing gains in both procedural and reflective knowledge (Beidas et al., 2014). Future research should explore how knowledge tests and behavioral rehearsal may be used together to evaluate knowledge gains across declarative, procedural, and reflective systems. Unfortunately, this study was also unable to establish how the didactics, demonstrations, and experiential exercises employed in the training differentially affected gains in declarative knowledge. Future research should focus on further establishing the relation between training strategies and therapist knowledge gain in particular knowledge systems. The present study also lacks information about therapist application of CBT following training, thus we are unable to connect knowledge gain that occurred during workshop training to therapist CBT implementation. An additional limitation of the study concerns generalizability, as therapists volunteered for the training and this may have impacted their level of engagement. Finally, although therapists’ knowledge increased after training, this study does not address the maintenance of CBT knowledge gain over time. Future research should explore the maintenance of CBT knowledge across all three systems (declarative, procedural, and reflective), as well as how maintenance of knowledge influences sustained CBT implementation.
In sum, a one-day CBT for depression workshop was indeed sufficient for increasing community therapist declarative knowledge about CBT. However, therapists maintained incorrect knowledge about several key CBT knowledge topics despite workshop training. These results have important implications for the planning and executing of CBT implementation efforts in community mental health settings, especially with respect to targeting specific knowledge systems that will maximize the effects of training efforts and enhance the provision of CBT in community settings.
References
- Beidas RS, Cross W, Dorsey S. Show me, don't tell me: Behavioral rehearsal as a training and analogue fidelity tool. Cognitive and Behavioral Practice. 2014;21(1):1–11. doi: 10.1016/j.cbpra.2013.04.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bennett-Levy J. Therapist skills: A cognitive model of their acquisition and refinement. Behavioural and Cognitive Psychotherapy. 2006;34(01):57–78. [Google Scholar]
- Bennett-Levy J, McManus F, Westling BE, Fennell M. Acquiring and refining CBT skills and competencies: which training methods are perceived to be most effective? Behavioural and Cognitive Psychotherapy. 2009;37(05):571–583. doi: 10.1017/S1352465809990270. [DOI] [PubMed] [Google Scholar]
- Borntrager C, Chorpita B, Higa-McMillan C, Weisz J. Provider attitudes toward evidence-based practices: are the concerns with the evidence or with the manuals? Psychiatric Services. 2009;60(5):677–681. doi: 10.1176/ps.2009.60.5.677. [DOI] [PubMed] [Google Scholar]
- Butler AC, Chapman JE, Forman EM, Beck AT. The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clinical Psychology Review. 2006;26(1):17–31. doi: 10.1016/j.cpr.2005.07.003. [DOI] [PubMed] [Google Scholar]
- Cross WF, Pisani AR, Schmeelk-Cone K, Xia Y, Tu X, McMahon M, Gould MS. Measuring trainer fidelity in the transfer of suicide prevention training. Crisis: The Journal of Crisis Intervention and Suicide Prevention. 2014;35(3):202–212. doi: 10.1027/0227-5910/a000253. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Davis D, O'Brien MAT, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? Jama. 1999;282(9):867–874. doi: 10.1001/jama.282.9.867. [DOI] [PubMed] [Google Scholar]
- De Jong T. Cognitive load theory, educational research, and instructional design: some food for thought. Instructional Science. 2010;38(2):105–134. [Google Scholar]
- Gibbons CJ, Fournier JC, Stirman SW, DeRubeis RJ, Crits-Christoph P, Beck AT. The clinical effectiveness of cognitive therapy for depression in an outpatient clinic. Journal of Affective Disorders. 2010;125(1):169–176. doi: 10.1016/j.jad.2009.12.030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gleacher AA, Nadeem E, Moy AJ, Whited AL, Albano AM, Radigan M, Hoagwood KE. Statewide CBT training for clinicians and supervisors treating youth: the New York State evidence based treatment dissemination center. Journal of Emotional and Behavioral Disorders. 2011;19(3):182–192. doi: 10.1177/1063426610367793. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jensen-Doss A, Cusack KJ, de Arellano MA. Workshop-based training in trauma-focused CBT: An in-depth analysis of impact on provider practices. Community Mental Health Journal. 2008;44(4):227–244. doi: 10.1007/s10597-007-9121-8. [DOI] [PubMed] [Google Scholar]
- Maunder L, Milne D, Cameron L. Pilot evaluation of brief training in CBT for primary care practitioners. Behavioural and Cognitive Psychotherapy. 2008;36(03):341–351. [Google Scholar]
- McCall H, Arnold V, Sutton SG. Use of knowledge management systems and the impact on the acquisition of explicit knowledge. Journal of Information Systems. 2008;22(2):77–101. [Google Scholar]
- Merrill KA, Tolbert VE, Wade WA. Effectiveness of cognitive therapy for depression in a community mental health center: A benchmarking study. Journal of Consulting and Clinical Psychology. 2003;71(2):404. doi: 10.1037/0022-006x.71.2.404. [DOI] [PubMed] [Google Scholar]
- Myles PJ, Milne DL. Outcome evaluation of a brief shared learning programme in cognitive behavioural therapy. Behavioural and Cognitive Psychotherapy. 2004;32(02):177–188. [Google Scholar]
- Nakamura BJ, Higa-McMillan CK, Okamura KH, Shimabukuro S. Knowledge of and attitudes towards evidence-based practices in community child mental health practitioners. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(4):287–300. doi: 10.1007/s10488-011-0351-2. [DOI] [PubMed] [Google Scholar]
- Shafran R, Clark DM, Fairburn CG, Arntz A, Barlow DH, Ehlers A, Ost LG. Mind the gap: Improving the dissemination of CBT. Behaviour Research and Therapy. 2009;47(11):902–909. doi: 10.1016/j.brat.2009.07.003. [DOI] [PubMed] [Google Scholar]
- Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don't train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology. 2005;73(1):106. doi: 10.1037/0022-006X.73.1.106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Simons AD, Padesky CA, Montemarano J, Lewis CC, Murakami J, Lamb K, Beck AT. Training and dissemination of cognitive behavior therapy for depression in adults: a preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology. 2010;78(5):751. doi: 10.1037/a0020569. [DOI] [PubMed] [Google Scholar]
- Simons AD, Rozek DC, Serrano JL. Wanted: Reliable and Valid Measures for the Science of Cognitive Behavioral Therapy Dissemination and Implementation. Clinical Psychology: Science and Practice. 2013;20(2):181–194. [Google Scholar]
- Westbrook D, Sedgwick-Taylor A, Bennett-Levy J, Butler G, McManus F. A pilot evaluation of a brief CBT training course: impact on trainees' satisfaction, clinical skills and patient outcomes. Behavioural and Cognitive Psychotherapy. 2008;36(05):569–579. [Google Scholar]
