Web-Based Therapist Training on Cognitive Behavior Therapy for Anxiety Disorders: A Pilot Study

Abstract

The need for clinicians to use evidence-based practices (such as cognitive behavior therapy [CBT]) is now well recognized. However, a gap exists between the need for empirically based treatments and their availability. This is due, in part, to a shortage of clinicians formally trained on CBT. To address this problem, we developed a Web-based therapist CBT training program, to increase accessibility to this training. The program uses a two-step approach: an interactive multimedia online tutorial for didactic training on CBT concepts, followed by live remote observation through a videoconference of trainees conducting CBT, with immediate feedback in real time during critical moments to enhance learning through iterative guidance and practice. Thirty-nine clinicians from around the county completed the online didactic training and 22 completed the live remote training. Results found a significant increase in knowledge of CBT concepts and a significant increase in clinical skills, as judged by a blind rater. User satisfaction was high for both the online tutorial and the videoconference training. Utilization of CBT by trainees increased after training. Results support the acceptability and effectiveness of this Web-based approach to training.

The need for clinicians to use evidence-based practices (EBPs) has been stressed from scientific, ethical, and marketing perspectives (Whaley & Davis, 2007). The Code of Ethics for all the major mental health professional organizations stresses the need for clinicians to use effective treatments and to integrate research evidence into their clinical practice. The American Psychological Association (APA) 2005 Presidential Task Force issued a policy statement advocating EBPs (APA Presidential Task Force on Evidence-Based Medicine, 2006), and evidence-based treatment guidelines are now under development (Clay, 2011). Systems for evaluating and classifying levels of empirical evidence have been proposed (APA, 2002) as well as the importance of evaluating both the efficacy of a treatment (“… the systematic and scientific evaluation of whether a treatment works”) and the effectiveness of a treatment (“..the applicability, feasibility, and usefulness of the intervention in the local or specific setting where it is to be offered”) (APA, 2002, p. 1053).

Research over the past several decades supports both the efficacy and effectiveness of cognitive behavior therapy (CBT) (one type of EBP) for anxiety disorders (Butler, Chapman, Forman, & Beck, 2006; Mitte, 2005; Norton & Price, 2007; Steven, 1996; Warren & Thomas, 2001). However, the availability of clinicians offering CBT and other EBPs remains limited (Goisman, Warshaw, & Keller, 1999; Shafran et al., 2009; Stewart & Chambless, 2007; Stewart, Chambless, & Baron, 2012) and, when offered, is often not delivered properly. (Kessler, Merikangas, & Wang, 2007; Stobie, Taylor, Quigley, Ewing, & Salkovskis, 2007). Reasons for the gap between the availability of these evidence-based treatments and their utilization by clinicians in community settings are the subject of much discussion. The shortage of clinicians trained on CBT methodology has been identified as one possible reason for this gap (Shafran et al., 2009).

The APA has made a concerted effort to remediate this shortage (APA, 1993). APA guidelines for accrediting programs in professional psychology state that programs should ensure interns are able to demonstrate knowledge of EBP theories and methods and demonstrate skill in formulating and implementing EBP intervention strategies (APA, 2007). A survey to examine the extent to which APA approved clinical programs were providing didactic instruction or supervised practicum training on a list of 26 EBPs found the average EBP treatment was included in didactic courses in 46% of graduate programs (range: 0%–96%) and in supervised practicum training in 44% (range: 0%–92%) (Crits-Christoph, Frank, Chambless, & Brody, 1995; Task Force on Promotion and Dissemination of Psychological Procedures, 1995). A follow-up study 10 years later found the numbers for didactic training had increased and the numbers for applied clinical training had decreased (Woody, Weisz, & McLean, 2005). A similar study found about one-third of programs in psychology and social work required both didactic instruction and clinical supervision in at least one EBP (Weissman et al., 2006). Research on new ways of disseminating clinician training in EBP has been suggested as an important initiative in helping facilitate access to EBPs by patients (National Institute of Mental Health, 2007; US Department of Health and Social Services, 1999).

New technologies provide a unique opportunity to increase accessibility to this specialized training by clinicians (Fairburn & Cooper, 2011), as well as to improve the quality of the training. A recent issue of this Journal reviewed several approaches using new technologies in psychotherapy supervision and training, and recommended continued research on how they may improve psychotherapy effectiveness and outcomes (Barnett, 2011). Web-based training can be available at any time to any clinician with Internet access. It obviates the need for travel (often at great distance) to the limited number of centers that offer training in CBT or other EBPs, and is not constrained by enrollment limitations due to class size or trainer availability. Trainees are not bound by time limitations and can work at their own pace (a recent study found time and cost to be the strongest predictors of unwillingness to obtain training on EBP) (Stewart et al., 2012). The quality of the training is enhanced through the use of interactive exercises and multimodal learning techniques, both of which have been found to increase knowledge retention (Gardner, 1993). Studies have shown Web-based therapist training to be more effective than paper-based manualized training alone (i.e., training through reading paper-based instruction manuals without additional clinician input) (Sholomskas & Carroll, 2006) and, in one study, superior to live training (Dimeff et al., 2009). Recent research using Web-based technologies to train clinicians on conducting clinical assessment found a combination of new technologies (i.e., multimedia, interactive, Web-based tutorials for didactic training combined with live remote observation of clinical skills through videoconference) is effective, can significantly improve clinical skills, and has significant advantages over traditional training programs (Kobak, Lipsitz, Williams, Engelhardt, & Bellew, 2005; Kobak, Engelhardt, & Lipsitz, 2006).

The goal of the current study was to develop and empirically evaluate a multimedia Web-based training program for mental health clinicians on how to provide effective cognitive– behavioral treatment to patients with anxiety disorders. It was hoped that, if successful, Web-based technologies could help increase the number of trained cognitive behavior therapists available to patients needing this treatment, thereby helping to fulfill the unmet needs in this area. Anxiety disorders are particularly appropriate for evaluating Web-based training on EBP, given the relatively clear stimulus–response link (Westen, Novotny, & Thompson-Brenner, 2004).

Web-Based Training Model

We propose a two-phase Web-based training model: (1) didactic training, through a Web-based interactive online tutorial, followed by (2) applied training, using Web-based videoconferencing. Applied clinical training is of particular importance, as the ability to actually administer the skills in clinical practice is often an overlooked aspect of training (particularly in continuing education). Although Web-based tutorials can be effective in increasing conceptual knowledge, didactic training alone does not ensure that a clinician will be able to apply these concepts in conducting effective CBT with clients (Sholomskas & Carroll, 2006). This two-stage Web-based clinical training model has been successfully implemented for training clinicians in the area of clinical assessment (Kobak et al., 2005; Kobak et al., 2006) and, in a modified form, providing CBT training for substance abuse counselors (modified to include Web group supervision vs. observation of actual practice) (Weingardt, Cucciare, Bellotti, & Lai, 2009). The current project examined the effectiveness of this model in training clinicians to conduct CBT in patients with anxiety disorders.

Method

Participants

Two cohorts of trainees were enrolled. Thirty-six trainees were initially enrolled and offered both the didactic training through online tutorial and the applied training using live videoconferencing. Owing to the large response and high level of interest, we allowed an additional 14 subjects to enroll and take the online tutorial only (study budget limitations precluded offering the additional subjects applied training as well). Thus, 50 subjects were enrolled (36 offered both didactic training and applied training and 14 offered the online didactic training only). Of these 50, 39 (78%) completed the online didactic training (28 from the didactic plus applied training cohort, 11 from the didactic training only cohort). Of the 28 subjects completing the didactic training from the didactic plus applied training cohort, 22 elected to proceed to the applied training (78%). Of these, all 22 completed the applied training (See Figure 1).

Figure 1.

Participant flow.

The 39 trainees completing the online tutorial consisted of nine male and 30 female subjects, including 29 social workers, two psychologists, and eight graduate students from social work or psychology. The mean age was 41.97 years (SD = 12.4, range: 24–75 years), and the mean years of clinical experience was 9.9 (SD = 10.2). Twenty-six (66%) were licensed to provide psychotherapy in the states they practiced. All report clinical experience, with 35 (90%) having an active clinical practice. Fourteen (36%) reported having received some type of previous CBT training (primarily continuing education lectures). Only 7 (18%) received training as part of their academic curricula, and only 2 (5%) were actually observed conducting CBT as part of their training. Twenty-nine (74%) reported using some CBT techniques in their practice. No differences were found between completers and non-completers on age (t(48) = 0.35, p = .723), gender (χ²(1) = .083, p = .774), education level (χ²(1) = 1.182, p = .277), previous CBT experience (t(48) = 1.104, p = .275), or years of clinical experience (t(48) = .135, p = .893).

Participants were recruited through a small print advertisement in the National Association of Social Work newsletter and through the Departments of Social Work and Psychology at the University of Wisconsin-Madison. Responses to the advertisement came from diverse regions in the country, including 16 different states in the continental United States, Hawaii, and Kuwait (one psychologist practicing overseas). The study was approved by the Allendale Institutional Review Board. Participants were compensated $200 for their time.

Procedure

After completing a pretest on their knowledge of CBT concepts, trainees were given a username and password to access the Web-based training. They then completed the online tutorial at their own pace. Trainees could e-mail the instructors with questions about the material. After completion of the tutorial, the trainees took a posttest of their knowledge of CBT concepts and two user satisfaction surveys. The online tutorial took approximately 5.5 hr to complete. Trainees were encouraged not to do the tutorial in one sitting, as studies have found spaced learning over time produces better long-term retention (Pashler et al., 2007). The program was able to bookmark where the trainee left off. The mean number of days it took to complete the tutorial was 13.5 days (SD = 7.3, range: 1–28 days).

After completing the online tutorial, 22 trainees (16 community therapists, 6 graduate students) received three remote applied training sessions with an experienced CBT trainer through video-conference. Before the first videoconference session, the study coordinator called each trainee to assist them in installing Skype, test the audio and video connection, and practice logging in and out of Skype sessions. Of the 22 trainees, 17 had a webcam and 14 were already Skype users. Those without webcams were provided an inexpensive ($40) Logitech webcam that could be installed instantly through the USB port (no software required) (http://www.logitech.com/en-us/webcam-communications/webcams). Skype software was downloaded automatically by clicking the “install” icon on the Skype Web site (http://www.skype.com/intl/en-us/get-skype/on-your-computer/windows). Skype sessions were initiated by the trainer, who clicked on the “video call” icon next to the trainee's name in their Skype address book. Skype automatically sends a message to the trainee that appears on their desktop, with a link to click to answer the call. Of 66 Skype training sessions, five were interrupted due to dropped calls (likely due to slower speed on the network due to network traffic).

During each training session, the trainer portrayed a standardized patient with a specific anxiety disorder (i.e., social anxiety disorder) while the trainee role-played as the therapist. The session began with the “patient” describing who she was, a little about her background, and the types of problems she was having. The therapist trainee then conducted CBT with the patient, focusing on two specific CBT techniques: cognitive restructuring (i.e., downward arrow, challenging likelihood overestimation, decatastrophizing) and exposure therapy (creating a fear hierarchy, preparing and assigning exposure tasks, exposure debriefing). The trainer provided feedback in real time both after and sometimes during each role-play therapy session. The use of videoconferencing allowed for immediate feedback in real time during critical moments, enhancing the potential to learn through iterative guidance and practice. Video also creates a greater “social presence” than audio alone affords (Cukor et al., 1998), which may help enhance the supervision alliance. Each training session was about 1 hr long. Participants completed a separate satisfaction questionnaire for the applied videoconference training.

The CBT training was conducted by a licensed clinical psychologist, who was in her second year of postdoctoral training, from the Anxiety Disorders Research Center at the University of California-Los Angeles. The trainer had extensive experience in providing CBT supervision and training as part of her graduate and postdoctoral experience.

Online Tutorial Content

The online tutorial consisted of nine modules covering the theory and practice of CBT for anxiety disorders: (a) Welcome and Introduction, (b) Principles of CBT, (c) Teaching Clients about the Nature of Anxiety, (d) Explaining Treatment Rationale to Clients, (e) Teaching Patients Self-Assessment Skills, (f) Helping Clients Develop a Fear Hierarchy, (g) Teaching Clients Breathing Techniques, (h) Managing Anxious Thinking (Cognitive Restructuring), and (i) Exposure Therapy. A more detailed description of the contents in each module is provided in Table 1. Longer modules are broken into smaller segments, which permit processing of smaller chunks of information at a time, avoiding cognitive overload and improving learning retention.

Table 1. Contents of the Online Tutorial.

Module	Description
1. Welcome and introduction	A video introduction by Dr. Michelle Craske welcoming the trainees to the course, and providing an overview of the course content and objectives (5 min).
2. Principles of CBT	The theoretical underpinnings of CBT for anxiety disorders, including (a) cognitive therapy (understanding the role of one's thoughts in determining one's emotions and behaviors), (b) classical conditioning (learning by association, where an innately aversive event can contribute to the development of a fear of objects, situations, or people associated with that event), and (c) operant conditioning (a process where a voluntary behavior that is increased or decreased depending on the environmental consequences that follow it). How each of these theories applies to case conceptualization and the development of anxiety disorders and to specific CBT treatment techniques is reviewed. The importance and nature of the therapeutic relationship in CBT is also reviewed, as well as the roles of both the therapist and client in CBT, and how to enhance the therapeutic alliance (20 min).
3. Teaching clients about the nature of anxiety	How to teach clients about the nature and purpose of anxiety, the difference between normal anxiety and anxiety disorders, and the function of somatic symptoms of anxiety (e.g., heart rate, blood flow, breathing, increased sweating). The latter is particularly important in the treatment of panic disorder, where fear of the symptoms themselves become an integral part of the fear cycle. Providing information on the nature and purpose of these symptoms helps clients counter catastrophic interpretations of these symptoms. The role of thinking in the etiology of anxiety and anxiety disorders is also reviewed (10 min).
4. Explaining treatment rationale to clients	Training therapists how to help clients understand the rationale behind treatment is a critical skill, as both treatment outcome and compliance are directly related to how much sense a treatment makes to the client, and that the more sense a treatment makes to the client, the more likely it is that they will comply with it (Raue, Schulberg, Heo, Klimstra, & Bruce, 2009). This module reviews how to explain the treatment rationale and the basic principles of CBT to clients, and the importance of clients' understandings of these principles in facilitating successful treatment (15 min).
5. Teaching patients self-assessment skills	Teaching clients the purpose and importance of self-assessment; the techniques for objectively recording their anxiety; how to maintain an anxiety diary; and how, what, and when to record. Self-assessment is used both before and during treatment to set treatment goals, develop a fear hierarchy, understand the context in which one's fears are developed and maintained (i.e., a “functional analysis”), and measure treatment outcomes. Objective self-monitoring helps clients gain a sense of control; improve their understanding of their own thoughts, feelings, and behaviors; and help them see the changes they are making through therapy (10 min).
6. Helping clients develop a fear hierarchy	Explaining to clients the rationale for confronting their fears, the role of avoidance in maintaining anxiety, and step-by-step guidance for creating a fear hierarchy. Providing a rationale to clients for overcoming avoidance is critical for successful treatment. If clients understand how avoiding their feared situations maintain and exacerbate their anxiety, they will be more motivated to conduct what are sometimes difficult exposures sessions.
7. Teaching clients breathing techniques	The physiology and mechanics of normal breathing and anxious breathing, the physiological changes associated with overbreathing (increased oxygen retention, changes in the oxygen–carbon dioxide balance; alkaline blood; vasoconstriction) and the physical symptoms that this overbreathing causes. The importance of breath control, its use as a therapeutic intervention in CBT, and step-by-step instructions for teaching diaphragmatic breathing to clients, including troubleshooting potential problems, typically encountered by clients (30 min).
8. Managing anxious thinking (cognitive restructuring)	Teaching clients the impact of their thoughts on their emotions and behaviors; the reciprocal relationship between thoughts, feelings, and behavior; and the two types of thinking errors most associated with anxiety disorders: (a) overestimating the likelihood of negative events and (b) overestimating the negative consequences of events (10 min).
	Teaching clients how to examine their thoughts on a detailed level and identify their underlying fears using the “downward arrow” technique (i.e., using “Socratic questioning” to ask appropriate questions so that clients end up “owning” the discovery or insight themselves, with no pressure to accept the therapist's point of view). How to teach clients to maintain a thought diary (20 min).
	Teaching clients to challenge their negative thinking. Once the underlying fears are identified using Socratic questioning, thought challenging is used to evaluate the evidence for and against the underlying fears, followed by re-evaluating the likelihood of their fears in light of the new evidence, and generating alternative explanations based on the new evidence. Use of the challenging anxious thinking form (20 min).
	Troubleshooting issues and concerns often encountered by patients when doing cognitive restructuring such as “What if my clients resist thought challenging due to fear that focusing on what worries them will make them feel worse”; “Is it OK for me to provide reassurance that the feared negative event is unlikely to happen?”; “What if clients get discouraged because cognitive restructuring does not immediately decreased their anxiety?”; What if a client acknowledges the chances of the feared event is extremely remote, yet feels the consequences if it did occur would be intolerable?” (10 min).
9. Exposure therapy	The goals of exposure therapy (i.e., to learn directly, by experiencing the feared situation, whether or not the feared outcome occurs); the critical factors in exposure therapy (i.e., new learning, not anxiety reduction per se); the therapist's role in exposure therapy (i.e., to help set goals and design practices, provide encouragement, support and feedback, and serve as a model in some cases); and preparing the client for exposure therapy by providing a rationale for exposure treatment (10 min).
	Steps in designing exposure practice; choosing goals from the fear hierarchy; identifying underlyingthoughts and hypotheses to be tested; establishing the most effective conditions for successful exposure; and deciding on rate and degree of exposure, and the length, frequency, and timing of exposure sessions; (20 min).
	Preparing clients for their initial exposure by thought challenging the main worry before exposure; planning ways of managing anxiety during exposure practice; identifying forms of avoidance during exposure practice and planning ways of managing avoidance; and reviewing the practice with anxious situations form (20 min).
	Reviewing results of exposure sessions and evaluating what was learned; reviewing cognitions during exposure, and thought challenging them to learn from the experience; troubleshooting issues encountered, including: (a) noncompletion of any exercise, (b) escape during exposure, (c) continued high anxiety levels after many repetitions, (d) client discouragement due to fluctuation in performance, and (e) dealing with clients' negative thinking about their performance (20 min).
10. Enhancing client motivation	One of the therapists' tasks is to create a set of conditions that will enhance the clients' own motivation for change. Clients with anxiety disorders often have mixed emotions about change, particularly about exposure therapy, as avoidance behavior has served, at least in part, to successfully minimize their anxiety. This ambivalence is an important part of the change process. Exploring motivation and ambivalence fore initiation of CBT for anxiety greatly improves treatment outcomes. This module provides a more thorough review of techniques from motivational interviewing, including exploring ambivalence, developing discrepancy, dealing with resistance, and enhancing self-efficacy.

There was also a module on enhancing client motivation, that is, how to motivate clients with anxiety disorders using techniques from motivational interviewing, including exploring ambivalence, developing discrepancy, dealing with resistance, and enhancing self-efficacy (see Table 2). Recent research has found enhanced outcomes for patients with anxiety disorders when motivational interviewing techniques are added to CBT therapy (Westra, Arkowitz, & Dozois, 2009; Westra & Dozois, 2008; Westra & Dozois, 2006). Social anxiety disorder was used as an example to illustrate conceptual points.

Table 2. Examples From Motivational Interviewing Module.

Concept	Sample text from module
1. Exploring ambivalence	One of the therapist's tasks is to create a set of conditions that will enhance the client's own motivation for change. Clients with anxiety disorders often have mixed emotions about change, particularly about exposure therapy, as avoidance behavior has served, at least in part, to successfully minimize their anxiety. This ambivalence, rather than being seen as an obstacle to change, is actually an important part of the change process. It is important that the therapist take a nonpejorative view of this ambivalence. Recent studies have shown that exploring motivation and ambivalence before initiating CBT for anxiety disorders greatly improves treatment outcomes. Thus, it is important for therapists to normalize this ambivalence, and to point out that change involves not only action but also an equally important process of thinking about the pros and cons of taking action, and weighing the costs and benefits. One simple way to help clients explore their ambivalence is to have client do a cost–benefit analysis, listing good and bad things about their current behavior, and the pros and cons for and against change.
	Good things about current behavior By avoiding the situation, I can manage my anxiety and make things more predictable. Not so good things about current behavior I'm missing out on many things I want to do.	Good things about changing behavior I'll be able to go to movies again and other things I currently avoid and I'll be able to fly again. Not so good things about changing behavior I'll have to experience a lot of anxiety doing the exercises and will give up my old ways of control.
2. Developing discrepancy	In developing discrepancy, the therapist helps the client explore the discrepancy between the way things are and the way the client would like them to be. This helps clients become “unstuck,” as it helps them evaluate how important change is to them. Motivation for change only occurs when people perceive a discrepancy between where they are and where they want to be. This can be facilitated by asking the client questions such as “What are the things that are most important to you in life?” and “If things worked out well for you, what would this look like?” Follow this by asking how their current behavior fits or does not fit with the way they want things to be. Ideally, it is important for the client to provide the pros and cons for change, not the therapist. However, sometimes the therapist can, if done correctly, raise the client's awareness of the personal consequences of their current behavior. For example, one may say to an agoraphobic client, “You mentioned that being a good parent is very important to you. How does someone who values close family relationships feel about not being able to attend your child's music recitals or parent-teacher conferences?” When this is done correctly, it is the client, and not the therapist, who voices the arguments for change.
3. Dealing with resistance	Resistance in therapy can take many forms, including denying the problem (i.e., unwillingness to recognize problems, cooperate, or accept responsibility), arguing with the therapist, interrupting the therapist, or ignoring the therapist. Resistance involves a combination of client ambivalence and how the therapist responds to it. Arguing for a change with a client is likely to push the client to argue against it. In general, resistance is not dealt with head on (such as by disagreeing or trying to persuade), but by either: (a) using reflective statements or, in some cases, (b) shifting focus. How the therapist responds to the resistance will influence whether resistance increases or decreases.
4. Supporting self-efficacy	Self-efficacy is the belief that one can perform a particular behavior or succeed at a particular task. Supporting the clients' belief that they can change is critical, because if they have little hope that they can change, there is little reason for them to face the problem. Supporting self-efficacy is particularly important once a client has sufficiently worked through his or her ambivalence about change and starts to make statements indicating he or she is considering action (i.e., “Christmas is coming and I would really like to get over my fear of flying so that I can visit my children”). It is important to switch into “action mode” once the client is ready; continuing to use primarily reflection and validation at this stage would impede the client's progress.

Online Tutorial Teaching Strategies

The online tutorial contained instructional material presented in a variety of formats: interactive exercises, animations, graphical illustrations, and videos of an expert clinician (Dr. Craske) demonstrating the techniques covered in each module with a mock client (see Table 3). “Challenge questions” were used as a way to both present new material and to reinforce learning (Roediger & Karpicke, 2006). Asking questions about material not yet learned is a powerful way to involve learners and enhance retention (Roediger & Karpicke, 2006). The tutorial was designed to maximize interactivity, which studies have found engages students in the learning process and enhances knowledge retention (Gardner, 1993; Vincent & Ross, 2001). Principles of instructional design were used to “chunk” material in appropriate segments (based on limits of working memory) and to guide the presentation of material in ways that enhanced learning (Mayer & Moreno, 2003).

Table 3. Examples of Instructional Design From Online Tutorial.

Type	Description
1. Interactive exercise	Reviewing the steps in cognitive restructuring. Trainees drag each clinical intervention to their correct order on the right, and then press “display answers” to see if their answer was correct. Go to www.CBTTherapistTraining.com/Slide01.htm to view functioning version of this slide
2. Interactive exercise	Teaching diaphragmatic breathing. Trainees click on the stopwatch to time their breathing and practice this skill. Audio says “Breathe at a rate of around 10 breaths per minute. By counting to 3 when inhaling and when exhaling, each breath will take about 6 seconds. Breathe smoothly from the diaphragm without taking big breaths. Practice your breathing by clicking the stopwatch.” Go to www.CBTTherapistTraining.com/Slide02.htm to view functioning version of this slide.
3. Animation	Explaining the nature and purpose of anxiety. Graphic of shield fading to graphic of caveman defending against an animated charging lion. Audio says “Anxiety is a natural human emotion that everyone feels sometimes. It is not a bad thing in and of itself, as it serves as useful function: to help protect us from danger. Anxiety has its evolutionary roots back in the days when we were hunters and gatherers, in order to help us deal with dangers from our ancient past”. Go to www.CBTTherapistTraining.com/Slide03.htm to view functioning version of this slide.
4. Graphical illustration	Explaining the function of somatic symptoms of anxiety. Graphic of anatomical illustration of the human body. Instructions say “Click each body part for a more detailed explanation of what happens when you become anxious.” Slide introduced with the following audio: “Anxiety increases our heart rate so that we can get more oxygen to our muscles in order to help us either fight or run away (fight or flight); it causes our breathing to become faster in order to increase oxygen levels in the blood which facilitates fighting or fleeing; it causes our blood to flow away from our extremities in case we are wounded, to reduce bleeding. All of these things are normal physical reactions. While they may not feel pleasant, they are not in and of themselves harmful. They are simply a sign that the body is preparing to take action because it senses a threat.” Go to www.CBTTherapistTraining.com/Slide04.htm to view functioning version of this slide.
5. Graphical illustration	Troubleshooting issues in cognitive restructuring. Vignettes are used to portray common questions CBT trainees often ask, and their answers. In this case, a cartoon vignette of a therapist in training asks his supervisor whether it is appropriate, as part of thought challenging, to reassure his clients that their feared negative outcomes are unlikely to happen. The supervisor answers and provides a rationale for the answer. Go to www.CBTTherapistTraining.com/Slide05.htm to view functioning version of this slide.
6. Challenge question	Physiology of breathing in anxiety. Challenge questions ask about material not yet covered as a way to learn the new material. Trainees click on their choice, and are then given feedback and the rationale for the correct answer. In this example, trainees are asked “Which of the following is a problem from breathing too much?” Three response options are provided. Go to www.CBTTherapistTraining.com/Slide06.htm to view functioning version of this slide.

Outcome Measures

Effectiveness

Online tutorial

Improvement in trainees' knowledge of CBT concepts was evaluated using a 38-item multiple-choice pre- and posttest of the items covering the tutorial content. Items were developed by domain experts (Drs. Craske, Rose, and Kobak) to ensure they accurately reflected tutorial content. Items covered not only trainees' understanding of CBT concepts and procedures but also their understanding of common difficulties encountered and how to address them (Fairburn & Cooper, 2011). No feedback was given to trainees after the pretest as to the correct answers. Coefficient alpha of the test was .73 in our sample.

Applied training

Improvements in clinical skill were evaluated using the Yale Adherence and Competence Scale (YACS) (Carroll et al., 2000; Nuro et al., 2000). The YACS is a well-validated scale for rating therapist adherence and competence in delivering cognitive–behavioral treatments for substance use disorders. It has good inter-rater reliability (intraclass correlation [ICC] = .88) and internal consistency reliability (coefficient alpha: r = .89 in our data) and correlates well with measures of therapist alliance (Carroll et al., 2000). A modified version of the cognitive–behavioral treatment subscale was used. Each YACS item was rated on a 7-point scale ranging from “very poor” (1) to “excellent” (7) (see Table 4). The YACS manual provides guidelines for rating each anchor point. The scale contained three items evaluating therapist skill in conducting cognitive restructuring and three items evaluating ability to conduct exposure therapy. All training sessions were audiotaped, and the first and last sessions were rated by an experienced CBT clinician. To minimize expectancy bias, the rater was blinded to whether it was the initial training session or final session. Tapes were prescreened to edit out any comments that would identify session number. Studies have shown that knowing the temporal order of sessions significantly impacts raters' judgments of improvement (Quinn et al., 2002). In addition, whenever the trainer “broke role” to provide feedback, the trainer stopped the tape recording so that feedback would not be included in the tape recording, and thus not able to bias the tape raters scoring. Because the trainee received feedback during the initial training session, the first session is not a true baseline. The subsequent session is a more accurate test as to whether the skills from the previous session were consolidated and whether, on the whole, the trainee is improving.

Table 4. Mean Item Scores and Effect Sizes: Yale Adherence and Competence Scale.

Item	Mean (SD) initial sessiona	Mean (SD) final sessiona	t	p	Effect size (d)	95% CI
1. Identified client's thoughts and beliefs using downward arrow technique	4.0 (1.8)	4.2 (1.4)	.35	.728	0.11	0.77, −0.55
2. Challenged odds of thought being true by using challenge questions, examination of evidence, and looking at alternative explanations	3.2 (1.4)	4.5 (1.3)	3.41	.003	0.94	1.52, 0.37
3. Decatastrophized with client by using different challenge questions and looking at alternative explanations/ways to look at and cope with situation	2.8 (1.2)	4.2 (1.2)	4.00	.001	1.15	1.75, 0.55
4. Worked collaboratively with client to create a fear hierarchy	3.4 (1.5)	4.5 (1.4)	3.21	.004	0.72	1.20, 0.27
5. Worked collaboratively with client to choose appropriate exposure exercise; went through how to conduct exposure at home using the form; reiterated important aspects of exposure	3.1 (1.5)	4.5 (1.3)	4.54	.000	1.03	1.51, 0.56
6. Helped client synthesize information learned from previous exposure and adequately addressed issues that came up if client did not complete exposure as planned	3.5 (1.6)	4.8 (1.4)	3.22	.004	0.84	1.39, 0.30
Mean item rating	3.3 (1.3)	4.4 (1.0)	3.87	.001	0.97	1.51, 0.45

^aScale: 1 = very poor, 2 = poor, 3 = acceptable, 4 = adequate, 5 = good, 6 = very good, 7 = excellent.

Tape ratings were conducted by an experienced CBT clinician from the University of California Los Angeles Collaborative Center for Integrative Medicine (a clinical psychologist with extensive experience in conducting CBT therapy with published research on CBT interventions). Although only a single rater was used to evaluate tapes, training and calibration on the YACS was done before the study to ensure that her ratings were reliable with those with Dr. Taylor. Six tapes were independently coded by each rater, and ratings and rationales for the ratings were discussed, until a consensus score was reached and a common understanding of the YACS guidelines was achieved.

User Satisfaction

Online tutorial

User satisfaction with the technical aspects of the online tutorial was assessed using the System Usability Scale (SUS) (Bangor, Kortum, & Miller, 2009; Brooke, 1996). The SUS is a reliable well-validated 10-item scale designed to evaluate the usability and user satisfaction with Web-based applications and other technologies. Each item is rated on a 5-point scale, with anchor descriptions provided for the end points (1 = strongly disagree, 5 = strongly agree). A global rating of user friendliness is also obtained. The SUS has good internal consistency reliability (coefficient alpha: r = .82 in our sample) in assessing usability across diverse types of user interfaces (e.g., Web, Interactive Voice Response [IVR], cell phone, etc.). The content of the SUS is shown in Table 5.

Table 5. Mean Ratings on System Usability Scale Items.

Item	Mean (SD) ratinga
1. I would use a tutorial designed like this again	4.7 (0.52)
2. I found the technical features of the tutorial unnecessarily complex	1.4 (0.67)
3. I thought the tutorial was easy to use	4.5 (0.60)
4. I think that I would need the support of a technical person to be able to use this tutorial	1.1 (0.48)
5. I found the various technical features in this tutorial were well integrated	4.4 (0.81)
6. I thought there was too much inconsistency between the technical features in this tutorial	1.5 (1.0)
7. I would imagine that most people would learn to use this tutorial very quickly	4.5 (0.82)
8. I found the technical features very cumbersome to use	1.5 (1.0)
9. I felt very confident using the tutorial	4.7 (0.46)
10. I needed to learn a lot of things before I could get going with this tutorial	1.2 (0.45)

^aRating scale: 1 = “Strongly disagree” – 5 = “Strongly agree.”

Descriptive data were also gathered on user satisfaction with the clinical content of online tutorial. Participants were asked to rate 15 dimensions of the user experience on a 4-point scale (strongly agree, agree, disagree, strongly disagree) (see Table 6). These items had been developed in previous research to gather descriptive information on user satisfaction with Web-based training (Kobak et al., 2006; Kobak, Opler, & Engelhardt, 2007). Coefficient alpha of the scale was .90 in our sample.

Table 6. User Satisfaction Questionnaire for Online Tutorial.

Question	Mean rating (SD)a
1. The objectives of the tutorial were clear	3.8 (0.41)
2. The tutorial was well organized	3.7 (0.46)
3. The material was presented in an interesting manner	3.5 (0.51)
4. There were sufficient examples and illustrations	3.7 (0.47)
5. The concepts were clearly presented and easy to understand	3.7 (0.46)
6. The video examples were helpful in illustrating the concepts	3.6 (0.59)
7. The animations and graphics were helpful in illustrating the concepts	3.5 (0.60)
8. The tutorial increased my knowledge about CBT for anxiety disorders	3.9 (0.37)
9. I feel capable of applying these techniques to my clients	3.4 (0.54)
10. The length of the tutorial was appropriate	3.4 (0.54)
11. This technology was as effective as traditional teaching methods in helping me learn the material	3.5 (0.60)
12. I would recommend this course to others	3.7 (0.47)
13. I enjoyed taking the tutorial	3.7 (0.47)
14. Overall, I was satisfied with this tutorial	3.7 (0.46)
Mean item rating	3.6 (0.32)

^aResponse options: 1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree.

Applied training

Descriptive data were also gathered on user satisfaction with the applied training. Participants rated their experience along 11 dimensions, covering both the clinical content of the training as well as their experience using the videoconferencing technology. Items were rated on a 4-point scale (strongly agree, agree, disagree, strongly disagree) (see Table 7). Coefficient alpha of the scale was .87 in our sample.

Table 7. Results From the Videoconference Training Satisfaction Questionnaire.

Question	Mean ratinga
1. The videoconference training was helpful in learning how to apply the CBT techniques	3.8
2. The videoconferencing got in the way of the learning experience	3.6b
3. I found the technical features of videoconferencing difficult to use	3.4b
4. Videoconferencing is a good way to receive training on CBT for those who live far from training sites	3.5
5. The videoconference training improved my ability to conduct CBT with clients	3.5
6. I feel capable of applying these techniques to my clients	3.4
7. Videoconferencing is as effective as face-to-face training in helping me learn how to conduct CBT	3.0
8. I would recommend this training to others	3.7
9. I enjoyed taking the videoconferencing training	3.6
10. Overall, I was satisfied with videoconference training	3.6
11. I feel less anxious with videoconference training than with face-to-face training	2.2
Mean item rating	3.4

^aResponse options: 1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree.

^bThese items were reverse scored, where a lower rating meant greater satisfaction and then transformed (i.e. 5 – x).

Use in clinical practice

We followed up with trainees through e-mail 2 months after completion of the training to evaluate the extent to which they were actually using CBT with their clients using the single question “How often do you use CBT with your clients?” (never, rarely, occasionally, often, very often, almost exclusively). The same question was asked before training.

Statistical Analyses

Mean change from pretest to posttest on the online tutorial and on the YACS scale was computed using paired t tests. In addition, standardized mean effect sizes (ESs) for the t tests were calculated using the formula d = t × sqrt (2 × ((1–r)/n) (Cohen, 1988) (statistical software provided by David B. Wilson, Ph.D., http://mason.gmu.edu/∼dwilsonb/ma.html, accessed through the APA Web site, http://www.apa.org/pubs/journals/pst/resources.aspx). ESs for t tests were considered large at .80, medium at .50, and small at .20 (Cohen, 1992).

Results

Web-Based Didactic Tutorial

Improvements in conceptual knowledge

We examined changes in pre- to posttest scores on knowledge of the concepts covered in the tutorial. We found a significant increase in the mean number of correct items pre- to post-tutorial, from 14.79 (SD = 4.12) to 32.21 (SD = 3.13), t(38) = 25.83, p < .001, ES (d) = 4.76, 95% confidence intervals (CIs) (5.13, 4.48). Thirty-one trainees (80%) scored 80% correct or better (i.e., 30 of 38 items correct) after taking the tutorial, compared with none before taking the tutorial.

User satisfaction with technical aspects of online tutorial

The mean total score on the SUS was 90.4 (SD = 10.92) (scale range is 0–100). This corresponds to a score of excellent on the SUS. Sixty-two percent of trainees had a mean score of 90 or more. Examination of individual SUS items found high levels of satisfaction with the technical aspects of the online tutorial. Participants thought the tutorial was easy to use, felt confident using it, thought that it was well-integrated, and agreed they would use a tutorial designed like this again (see Table 5). The mean score on the single-item global rating of user friendliness was 6.0 (excellent) (range = 1 [worst imaginable] to 7 [best imaginable]).

User satisfaction with clinical content of online tutorial

Ratings on the 14 individual User Satisfaction Questionnaire items are shown in Table 6. The mean item rating on user satisfaction with the content of the online tutorial was 3.6 (SD = .34), midway between agree and strongly agree (possible item range = 1–4), indicating a high level of satisfaction with the tutorial's clinical content. All participants agreed or strongly agreed that the tutorial increased their knowledge about CBT for anxiety disorders, that they enjoyed the tutorial, and that they would recommend the tutorial to others. All but one felt capable of applying the techniques with clients

Videoconference Applied Training

Improvements in applied clinical skills

The mean YACS total score improved from 19.9 (SD = 7.5) during the initial applied videoconference training session to 26.5 (SD = 6.0) at the final applied training session, t(21) = 3.86, p = .001, ES (d) = .97, 95% CIs (1.50, 0.46). This corresponds to an increase of roughly from the midpoint between acceptable (defined as slightly below the average level of competence) and adequate (defined as the competence level of the average clinician) to the midpoint between adequate and good (defined as slightly better than the average clinician). Significant improvement was also found on both subscales: the ability to conduct cognitive restructuring improved from 9.9 (SD = 3.9) to 12.8 (SD = 3.2), t(21) = 2.87,p = .009, ES (d) = .81, 95% CIs (1.38, 0.22), and the ability to conduct exposure therapy improved from 10.0 (SD = 3.9) to 13.7 (SD = 3.5), t(21) = 4.51, p < .001), ES (d) = 0.99, 95% CIs (1.49, 0.55). Improvements on individual YACS items are presented in Table 4.

Satisfaction with videoconference training

The mean item score on the Videoconference Satisfaction Scale (transposed for negatively scored items, where a lower rating meant greater satisfaction) was 3.4 (SD = .45), roughly midway between agree and strongly agree. Ratings on the 11 individual items are shown in Table 7. All participants agreed or strongly agreed with all items, with the exception of item 11 (“I felt less anxious with videoconference training than face to face training”). Overall, trainees found the experience to be useful and enjoyable, thought that it increased their ability to conduct CBT with patients, felt it was as effective as face-to-face training, and thought that it was a good way to learn CBT. None of the participants reported the technology interfered with the training experience. Global rating of user friendliness of the videoconference training was 5.9 (SD = .71) (just below excellent).

Use in Clinical Practice

Finally, we followed up with trainees 2 months after completing the training to see to what extent they were using CBT in their practice with patients. The percentage of trainees who reported using CBT with their patients often or very often increased from 45.5% before training to 75% after training.

Discussion

Several barriers to the dissemination of CBT have been identified (Shafran et al., 2009). The current study was aimed at ameliorating one of these barriers: the lack of CBT training available to therapists. Results of the current pilot study provide evidence in support of the effectiveness and feasibility of the Web-based training model. Further research using a control group and randomization is warranted to examine the efficacy of this training model. Effective training, like effective treatments, will not work unless people actually use them. Thus, feasibility in the form of user satisfaction is an important factor when developing training methodology. The current study found trainees had high levels of satisfaction with both the clinical content and the technical features of the training. Being able to obtain training online greatly increases accessibility and dissemination. The fact that the training was done by an experienced, but newly trained, psychologist (vs. a more highly tenured “thought leader” in the field) gives promise for increased dissemination of the applied training as well.

As our ultimate goal was to improve the dissemination of CBT treatments to patients, it was important that trainees not only learned the CBT skills but actually implemented them in their clinical practice with patients. Trainees reported using CBT techniques more frequently after training, with 75% reporting using CBT often or very often, compared with 45% before training. Long-term follow-up on maintenance of skills and continued utilization of CBT techniques was not conducted, and would be of interest in future studies.

We also hoped that improved training would ultimately improve the quality of the CBT therapy being delivered to patients, as several studies have found that the quality of CBT currently being delivered to patients is suboptimal (Kessler et al., 2007; Stobie et al., 2007). By using principles of instruction design and making the tutorial interactive and multimodal, we hoped to enhance learning and retention. Most importantly perhaps is the inclusion of live applied training through videoconference. The ability to observe and evaluate trainees actually conducting CBT is critical (Rehm, 1997), and often a missing component in training programs. In our sample of 39 trainees, 29 reported using some CBT techniques with their clients, yet only two trainees reported ever being observed conducting CBT as part of their training. It is interesting to note that the mean YACS score during the first the applied training was 3.3, roughly halfway between acceptable and adequate. One of the limitations of the current study is that it is unknown how much the online tutorial by itself improved clinical skills, as we had no pretraining baseline measure of applied skills and no postonline tutorial measure conducted solely for evaluation purposes (i.e., with no feedback provided during the session).

Another issue that needs further exploration is the number of applied sessions necessary. In the current study, three applied training sessions raised the mean item score from 3.3 to 4.3, or roughly from between the midpoint of acceptable and adequate, as previously defined, to the midpoint between adequate and good. Whether more applied videoconference sessions can further improve applied skills is unknown, as is the number of sessions required to obtain incremental improvements. It should be noted that, given the limited length of the course, the goal was to facilitate the acquisition of CBT techniques, not to produce full-fledged CBT therapists. We feel this more modest goal is acceptable, given the dearth of available CBT clinician training, and thus the alternative being receiving no training at all. Our goal was to put some tools in the hands of clinicians who currently are not using them (or are using them incorrectly). Although issues such as case conceptualization, the therapeutic relationship, timing of interventions, and strategic issues were included in the course, a more in-depth coverage of these issues and more clinical training and experience over time is required to build proficiency, as with any skill.

One limitation to the current findings is that because the feedback was given in real time, it is unknown whether the ratings were a measure of actual skills that have been consolidated or a measure of behaviors that were generated based on the immediate feedback given. In reality, the impact of the immediate feedback is best measured by their performance at the next session, and their overall improvement over time. Measurement of skills after completion of training with actual patients in their clinical practice is the ultimate test of effectiveness, and will be addressed in future studies (see later in the text).

Another limitation to the current findings is the fact that participants were paid to participate in the study. It is unclear to what extent this may have impacted compliance or satisfaction ratings, or both. Thus, the generalizability of the findings from this sample to a sample of unpaid trainees is unknown. Although it is possible that in a commercial setting there would be a similar level of motivation, given one has paid to purchase such training, further study is needed to evaluate whether the outcomes found in this pilot study generalize to this and other scenarios (such as an employment setting where the organization, not the individual, pays for the training).

Other shortcomings include the fact that although no feedback was given to participants after the pretest on conceptual knowledge, this does not negate the potential for practice effects (i.e., looking up the answers to these issues). Although some studies have found pretesting to increase learning (i.e., through repetition and cueing to subsequent information) (Dempster, 1997; Roediger & Karpicke, 2006), pretesting may have nonetheless presented a confound, unless an alternative posttest version is used. In addition, although calibration training was done on the YACS, no formal inter-rater reliability study was done using a second blinded rater. Finally, although no differences were found between completers and noncompleters on several demographic, educational, and clinical variables, other unmeasured differences between these groups may have implications on the generalizability of the findings (e.g., breadth of computer experience, degree of course relevance, competing time commitments, differences in self-efficacy, etc.).

Shortcomings of the current study will be addressed in a follow-up study currently underway funded by the National Institute of Mental Health (NIMH). Pretraining baseline measures of clinical skill will be obtained to evaluate the relative contributions of didactic and applied training to clinical competence. More training sessions will be included to evaluate the “dose-response” relationship between number of sessions and improvements in clinical skill. The tutorial will be expanded to include additional discussion and examples of CBT techniques in the context of most anxiety disorders (i.e., generalized anxiety disorder, panic disorder, obsessive-compulsive disorder, posttraumatic stress disorder, social anxiety disorder), as well as a library of video illustrations of an experienced therapist executing these techniques with patients with each of these disorders. Most importantly, the effectiveness of the training in community care will be examined by following trainees out in the real world after they have completed the training and examining outcomes in their patients being treated for an anxiety disorder. We will compare outcomes in this cohort with a sample of community therapists conducting treatment as usual.

Although the advent of Web-based psychotherapy training is still relatively new, the exponential rate of growth of new technologies provides unique opportunities to improve both the quality and accessibility of clinical training. It is critical that new training approaches be empirically examined as to their feasibility, efficacy, and effectiveness. As is the case with evidence-based treatments, empirical data should guide further research and use of these new technologies in training clinicians. Just as professional organizations stress the need for clinicians to use effective treatments, and to integrate research evidence into their clinical practice, there is a parallel need to stress that trainers use effective training methods, and integrate research evidence into how training is conducted.

Appendix

Transcript From Video Vignette of Therapist Illustrating Downward Arrow With Mock Patient.

Therapist. To start the process of learning to evaluate thoughts and determine whether they are supported by the evidence or not we need to really identify what those thoughts are. So this next step is a bit like being an investigator where you're trying to discover through a series of questioning what you're really thinking in a given situation. And it's important to do in each situation in which you become anxious because there's likely to be some differences. What you think in one situation may differ from what you think in another situation. So why don't we take this situation of you being at the lunch room with your co-workers. Is that okay?

Patient. Uh-huh.

Therapist. And let's start off by thinking what it is you're most worried about in that situation.

Patient. Um, I think what I worry about is someone asking me a question.

Therapist. Okay. So if somebody were to ask you a question, what would happen? What are you concerned about?

Patient. Well, the last time I was in that situation and someone asked me about my weekend I started to get anxious; I started to feel the physical symptoms: my heart rate, shortness of breath, I started to feel hot.

Therapist. Right, so some of the physical symptoms popping up.

Patient. Yeah.

Therapist. And what if you felt those physical symptoms what would happen next? What are you worried about happening?

Patient. Well I fear that I'm going to go blank, or that I won't be able to answer their question, or that the response I give will be somewhat, you know, scattered or won't be concise or intelligent sounding.

Therapist. Okay. and if that happened—then what?

Patient. I think I'd worry about what they'd think about me.

Therapist. And what do you fear that they might think?

Patient. I'd feel worried that they'd think I was weird or just unusual in some way.

Therapist. Okay. So that they might think you're weird or unusual in some way. And what would that mean if they thought that?

Patient. Um, well, I think they probably wouldn't want to be my friend or maybe speak to me and eventually I wouldn't have any friends and may be alone.

Therapist. So the end of the chain of thoughts, it sounds like, is that you'll be alone.

Patient. Right.

Therapist. Okay, okay. So that technique that we just went through—trying to peal down what if ‘this’ or what would that mean or where would that go is called a downward arrow technique where we're really trying to really pin down what you're most concerned about in a given situation. And you can see, right, that what you're most concerned about as you walk into that lunch room—what you're most concerned about ends up being com-pletely—you're alone. It makes sense that you'd become anxious, right?

Patient. Uh-huh.

Therapist. Because I think anybody who is thinking those kind of negative thoughts and anticipating that this is the beginning of becoming isolated and alone would feel anxious in that situation.

Patient. Right, right.

Therapist. So the anxiety is appropriate for what you're thinking. Our next goal is to evaluate that thinking and to see if it really is consistent with the evidence or not. But for now, I want you to continue that process of trying to pin down what is it that's lying at the bottom of this chain of thoughts every time you go into a situation where you become anxious so that we can understand the steps and the logic.

Patient. Okay.

Therapist. Good! Good job.