CBT competence in novice therapists improves anxiety outcomes

Abstract

Objective

This study explores the relationships between therapist variables (CBT competence, and CBT adherence) and clinical outcomes of computer-assisted CBT for anxiety disorders delivered by novice therapists in a primary care setting.

Methods

Participants were recruited for a randomized controlled trial of evidence-based treatment, including computer-assisted CBT, versus treatment as usual. Therapists (Anxiety Clinical Specialists; ACSs) were non-expert clinicians, many of whom had no prior experience in delivering psychotherapy (and in particular, very little experience with CBT). Trained raters reviewed randomly selected treatment sessions from 176 participants and rated therapists on measures of CBT-competence and CBT-adherence. Patients were assessed at baseline and at 6, 12, and 18 month follow-ups on measures of anxiety, depression, and functioning, and an average reliable change index was calculated as a composite measure of outcome. CBT-competence and CBT-adherence were entered as predictors of outcome, after controlling for baseline covariates.

Results

Higher CBT-competence was associated with better clinical outcomes whereas CBT-adherence was not. Also, CBT-competence was inversely correlated with years of clinical experience and trended (not significantly, though) down as the study progressed. CBT-adherence was inversely correlated with therapist tenure in the study.

Conclusions

Therapist competence was related to improved clinical outcomes when CBT for anxiety disorders was delivered by novice clinicians with technology assistance. The results highlight the value of the initial training for novice therapists as well as booster training to limit declines in therapist adherence.

Therapist competence in and adherence to delivering mental health treatment are important markers of the integrity of a therapeutic package⁽¹⁾. Competence refers to the skill of a clinician in delivering a particular therapeutic intervention and to act as an “agent of psychological change”⁽²⁾ whereas adherence indicates the degree to which therapists follow a prescribed manual without deviation⁽³⁾. Competence and adherence are distinct but related concepts. In order for therapists to be competent in a therapeutic modality they must be adherent to the intervention protocol, but adherence to the intervention provides no guarantee of competence⁽⁴⁾. The correlation between competence and adherence ranges from r= .5 to r=.85⁽⁵⁾. It is essential to measure both competence and adherence in treatment outcome research in order to understand their respective contribution to outcomes.

Investigations of the relationship between therapist competence and outcome provide contradictory evidence. On the one hand, higher competency is related to better outcomes for depression, generalized anxiety disorder, and panic disorder^(6–12) and better child-reported (though not parent- or clinician-reported) anxiety outcomes⁽¹³⁾. On the other hand, competence only predicted clinician-rated outcomes after adherence to the protocol and rapport were added as covariates in another study, and did not predict self-reported depression⁽¹⁴⁾. In addition, therapists whose panic disorder clients achieved higher than average improvement did not differ from therapists whose clients achieved lower than average improvement on competence⁽¹⁵⁾, and competence did not predict substance abuse outcomes⁽¹⁶⁾. Many of these studies use the Cognitive Therapy Scale⁽¹⁷⁾as their measure of cognitive behavioral therapy (CBT) competence.

Higher therapist adherence ratings predict subsequent improvements in depression⁽¹⁸⁾, especially early in treatment⁽³⁾, and substance abuse^{(19, 20)}. However, there are contradictory findings as well^{(21, 22)}. Furthermore, one study suggested that therapist adherence occurring early in treatment is not predictive of adherence later in treatment, and measures of adherence late in treatment do not predict depression outcomes⁽²³⁾.

A recent meta-analysis by Webb et al.⁽¹⁾ concluded that neither competence nor adherence were significant predictors of therapeutic outcome; however, there was large variability in the effect sizes across studies. Most prior research has relied on therapists who were already deemed highly competent in cognitive behavioral therapy by the study investigators^{(9, 24)} and who underwent extensive training to meet criteria for “study therapists.” This selection bias is likely to produce a ceiling effect, thereby mitigating a full analysis of the effects of therapist competence and adherence. With an increased emphasis on dissemination of evidence-based practice, it is important to understand the effect of competency and adherence in mental health providers, most likely with expertise in areas other than psychology who are likely to be delivering services in primary care practices⁽²⁵⁾. However, research to date has not examined competence and adherence in therapists who are non-expert CBT clinicians.

The current study examined competency and adherence of novice, non-expert clinicians who provided treatment in the Coordinated Anxiety Learning and Management (CALM) study, an effectiveness trial of a treatment for anxiety disorders in primary care. Full details on the methodology of the multi-site study can be found in Roy-Byrne et al.⁽²⁶⁾ and Sullivan et al⁽²⁷⁾. The CALM study provided a unique opportunity to examine competence and adherence in a relatively naïve sample of clinicians delivering behavioral health, typical of real world settings such as primary care. Based on the available literature, we hypothesized that competence and adherence would predict treatment outcome. In addition, given the novice sample of therapists, we explored the degree to which competence and adherence changed as a function of time (i.e., from initial training to the later stages of the study). Prior research shows that CBT therapists increase their competence and adherence with more experience^{(12, 28–34)}. However, it is unclear if the same pattern of improvement occurs with non-expert or novice clinicians. This is a relevant question since it pertains to the value of booster training following initial training in CBT for novice therapists.

Methods

Participants

Participants presented to 17 primary care facilities throughout the United States and were referred to the study by their primary care doctors. They were assessed by their study therapist for eligibility, and were required to have a principal anxiety disorder of panic disorder, generalized anxiety disorder, social anxiety disorder, or post-traumatic stress disorder to participate. Diagnoses for participants were obtained based on the initial assessment by a consensus between the expert psychiatrist and psychologists involved in the study. Participants were recruited between June 2006 and August 2008. They were all between the ages of 18 and 75 years-old, spoke either English or Spanish, and were determined to have at least moderate and clinically significant anxiety according to the Overall Anxiety Severity and Impairment Scale (OASIS)⁽³⁵⁾. Participants were excluded if they were currently in CBT, had unstable or life-threatening medical conditions, marked cognitive impairment, psychosis, bipolar I disorder, any drug abuse or dependence other than alcohol or marijuana abuse, or active suicidal intent or plan. The mean age of participants included in the analysis reported herein was 43.5 years (SD = 12.6); they were predominately female (68.6%) and Caucasian (79.1%).

Therapists

Therapists were selected to be relatively novice to the delivery of mental health treatment in order to increase the generalizability of the findings. The therapists, referred to as Anxiety Clinical Specialists (ACS), included 6 social workers, 5 registered nurses, 2 masters level clinicians, and 1 doctoral level psychologist. Only 9 of the 14 ACS had prior experience in delivering mental health services (mean years of experience =3.89), and none had expertise in treating anxiety disorders (mean years of experience =.21) or in CBT (mean years of experience=.57). Full details of the therapist training are available in Rose et al.⁽³⁶⁾. Essentially, training was organized at the University of California, Los Angeles, but was provided locally for each site by study psychologists (or by UCLA supervisors at one site). Training was conducted over 10 weeks with weekly or bi-weekly meetings that included readings, didactic presentations, quizzes, viewing of videotapes, role-plays, training cases, and ongoing in-person and telephone supervision. To receive certification as an ACS, each therapist completed six sessions with up to three patients, of which at least three sessions were rated by supervisors as indicative of proficiency (by scoring within 1 point of an established “gold-standard” average for a particular tape). Following certification, weekly group supervision was conducted either via telephone or in person.

CALM intervention

Participants were randomized to usual care (UC) or to the CALM intervention (ITV), which was comprised of computer-assisted CBT, medication management, or their combination. In ITV, a computer-assisted CBT program guided the ACS and the patient on generic modules (self-monitoring, psychoeducation, fear hierarchies, breathing retraining, and relapse prevention) and modules tailored to the most distressing/disabling anxiety disorder (cognitive restructuring and exposure for feared external cues and feared internal cues)⁽³⁷⁾. Therapists interacted directly with patients but used the computer program to provide the agenda for the session as well as information (reading material, exercises, and videos) to patients. Following baseline assessment, 482 of 503 patients (95%) in ITV had at least 1 intervention contact. Over the 12 months after study entry, patients had a mean (SD) of 7.0 (4.1) (median= 8) CBT visits and 2.24 (3.57) (median = 1) medication/care management visits. Of the total 482 patients, 166 (34%) had only CBT sessions, 43 (9%) had only medication/care management sessions, and 273 (57%) had both. 424 patients (88%) had all visits by 6 months. For more details see Stein and colleagues⁽³⁸⁾.

Measures

Therapist Competence and Adherence

Each visit with the ACS was digitally audio-recorded. Competence and adherence ratings were gathered from 176 participants; up to four sessions for each participant were rated, for a total of 259 ratings. The tapes were selected according to a stratified randomization procedure which considered therapy site, the ACS at the site, the patient treated by the ACS, and the session number. The randomization procedure was weighted so that sites with more clients had more tapes reviewed. The sessions that were rated ranged from session 1 to session 20, and the median session rated was 4. One session per participant (out of a possible four sessions per participant) was randomly selected for inclusion in the current analysis.

Measures of CBT-Adherence and CBT-Competence were created for this study. While it would have been possible to use the Cognitive Therapy Scale ⁽¹⁷⁾, this measure emphasizes the use of cognitive techniques whereas the intervention in this study emphasized behavioral techniques. Because of the unique characteristics of this study, in particular the use of computer-assisted CBT which guided both the therapist and the patient, we felt it necessary to create a measure that would be more sensitive to the treatment delivery format.

(a) CBT-Adherence

A set of adherence ratings were established specifically for this study. The adherence ratings were made separately for each module of CBT. The number of adherence items ranged from 2–11 items per module. Each item was rated on a 1–7 scale where 1 indicated that the ACS did not complete the targeted goal, 4 indicated the ACS somewhat completed the item, and 7 indicated that the ACS fully completed the item. Some examples are: “To what extent did the ACS cover the effect of avoidance,” and “To what extent did ACS cover ‘Mistakes in Anxious Thinking?’” The average Cronbach’s alpha for adherence was acceptable (α=.683).

(b) CBT-Competence

The competence ratings also were tied to the content of each CBT module. For example, the ACS was rated on the degree to which he/she helped clients to develop alternative appraisals of ambiguous situations in the cognitive restructuring module, and to the degree that he/she helped clients address difficulties in “getting going” if they failed to complete exposure homework. Competence was defined as the skill with which the ACS delivered the intervention and was rated on a 1–7 scale ranging from none to excellent. The number of CBT-competence items ranged from 1–7 per module. The average Cronbach’s alpha value for competence was acceptable (α=.74).

Independent Raters

Two independent raters listened to audiotapes of each session. Each rater was trained in cognitive-behavioral therapy for anxiety disorders (at least 4 years), completed reliability training, and met established reliability criteria. The latter involved rating 5 audio tapes that were previously rated by expert psychologists: the raters were required to match within one point of the expert’s ratings on CBT-Competence and CBT-adherence on three out of five audiotapes. A check for rater drift conducted about half way through the ratings revealed inter-rater agreement (within 1 point on a particular item) on 86.7% of items.

Outcome measures

(a)Brief Symptom Inventory(BSI) ⁽³⁹⁾

The BSI-12 included items from the anxiety and somatization scales, with each item rated on a 0 to 4 Likert scale (0 indicates not at all and 4 indicates extremely). The BSI demonstrated good test-retest reliability, as well as high correlations with its parent measure, the Symptom Checklist-90R⁽⁴⁰⁾. The measures were administered by assessors at the RAND Corporation blind to treatment condition and assessment occasion, at baseline, 6 months, 12 months, and 18 months. In the full sample, the Cronbach’s alpha at was .87, .90, .90, and .90 at each assessment, respectively.

(b) Anxiety Sensitivity Index(ASI)⁽⁴¹⁾

The ASI is a 16-item self-report measure of anxiety sensitivity, or beliefs that anxiety is harmful⁽⁴¹⁾. Each item is rated on a 1–5 point Likert scale (1 indicates very little and 5 indicates very much). It has demonstrated good reliability and it is factorially independent from other measures of anxiety⁽⁴²⁾. The Cronbach’s alpha in the full sample was .88, .91, .91, and .91 at each assessment, respectively.

(c) Patient Health Questionnaire(PHQ-9)⁽⁴³⁾

The PHQ-8 is an 8-item measure of depression severity (the item assessing suicidal ideation and intent from the PHQ-9 was dropped as the measure was administered by non-clinicians), with each item rated on a 0–3 Likert scale (0 indicates not at all and 3 indicates nearly every day). The PHQ-9 has demonstrated adequate reliability and validity⁽⁴⁴⁾. In the full sample, the Cronbach’s alpha was .86, .90, .91, and .90 at each assessment, respectively.

(d) Sheehan Disability Scale(SDS) ⁽⁴⁵⁾

The SDS is a 3-item measure of functional impairment that uses a 0–10 point rating scale (0 indicates not at all and 10 indicates extremely). It has demonstrated high internal consistency and construct validity⁽⁴⁶⁾. In the full sample, the Cronbach’s alpha was .83, .90, .91, and .90 at each assessment, respectively.

Therapist characteristics

Therapists completed a questionnaire which asked about their age, and number of prior years of: a) clinical experience, b) formal psychotherapy training, c) conducting psychotherapy, d) CBT, e) short term dynamic therapy, f) general supportive therapy, g) other therapeutic modalities, and h) anxiety disorders experience.

Data analysis

An a priori decision was made to include the following key patient demographic and diagnostic variables at baseline as potential covariates if they were significantly associated with outcome: education (total years of education), race (white vs nonwhite), number of chronic medical conditions, insurance status (covered by insurance versus not), marital status (married/living together versus single), age, gender, employment status (currently employed versus not), diagnosis of Generalized Anxiety Disorder (yes or no), diagnosis of Panic disorder (yes or no), diagnosis of Social Anxiety Disorder (yes or no), diagnosis of Post-Traumatic Stress Disorder (yes or no), diagnosis of Major Depressive Disorder (yes or no), and diagnosis of Dysthymia (yes or no). In addition, total number of CBT sessions attended was another possible covariate. The included covariates are reported in Table 1.

Table 1.

Competence and adherence as predictors of outcomes, above covariates

β values (and R2 change values as relevant) for covariates and competence/adherence
	MDD	Insurance	Gender	Outcome change: baseline-6 Months	Competence	Adherence
RCI from baseline to 6 months	.754**	.992*	--	--	−.007	--
	.766**	.985*	--	--	--	−.016
RCI from baseline to 12 months	−.330**	--	--	--	−.156* (.024)	--
	−.332**	--	--	--	--	−.081 (.007)
	−.273**	--	--	−.511**	−.199** (.040)
	−.276**	--	--	−.515**	--	−.037
RCI from baseline to 18 months	−.254**	−.139	−.080	--	−.229** (.047)	--
	−.245**	−.132	−.106	--	--	−.149 (.021)
	−.148*	−.004	−.100	−.377**	−.326 (.097)**	--
	−.137	−.011	−.155	−.385**	--	−.150

Note:

^*p<.05,

^**p<.01. Possible covariates included insurance status, gender, or diagnosis of Major Depressive Disorder (yes or no). Values in parentheses are R² change values from analyses using a hierarchical regression approach. Predictors of primary interest (CBT-competence and CBT-adherence) are shaded in grey.

In an effort to reduce Type I error, a single continuous Reliable Change Index (RCI)⁽⁴⁷⁾ variable was calculated that took into account scores on the BSI, ASI, PHQ-8, and SDS for each follow-up time point. The RCI is calculated by subtracting the time 1 score on a measure from the time 2 score on the measure, and dividing by a formula that takes into account the standard deviation on the measure and the reliability of the measure.

$$\text{RCI}=\frac{x2-x1}{\sqrt{[2s1{(\sqrt{(1}-rx))}^2]}}$$

*Note: This formula was taken from Jacobson and Traux⁽⁴⁷⁾

Here, x2 represents the score on the time 2 score, x1 represents the time 1 score, s1 represents the standard deviation, and rx represents the reliability of the measure in question. The standard deviation from the baseline measure of each outcome variable and reliability values that were used in this analysis are as follows: ASI SD =13.36, reliability=.75⁽⁴¹⁾; BSI SD = 8.64087, reliability =0.9⁽³⁹⁾; PHQ-8 SD = 6.08171, reliability = .94⁽⁴⁸⁾; SDS SD= 6.92900, reliability = .70⁽⁴⁹⁾. An RCI was calculated for each outcome measure, and then the four RCIs were averaged to obtain an overall RCI. According to Jacobson and Traux⁽⁴⁷⁾ and RCI of an absolute value greater than 1.96 represents significant change. Three total RCIs were calculated to represent: 1) change from baseline to six months, 2) change from baseline to 12 months, and 3) the change from baseline to 18 months.

In this analysis, competence and adherence ratings were nested within therapist. Therefore, an intraclass correlation (ICC) was calculated to determine the amount of within-therapist similarity. When this was calculated in relation to the RCI from baseline to 6 months, the ICC was 0.10185, which was large enough to justify analyzing data from this RCI within a multilevel modeling framework. In this case, RCI scores were nested within therapists using restricted maximum likelihood (REML) estimation. However, when the ICC was calculated for the RCI from baseline to 12 months and from baseline to 18 months, the ICC was small enough to suggest that an un-nested hierarchical linear regression approach would best represent the data (ICC= 0.00985 and 0.00402 for 12 and 18 months, respectively). In these cases, competence or adherence were entered into the second step of the model so that they could be examined over and above the influence of any relevant covariates.

In order to control for the temporal confound of symptom change predicting competence and adherence, a series of regressions were conducted which predicted competence and adherence from an average z score of the baseline symptom variables and from the RCI which included symptom change from baseline to 6 months. In addition, a series of secondary hierarchical linear regression analyses were conducted in which symptom change from 6 to 12 months and from 6 to 18 months were predicted after controlling for symptom change from baseline to 6 months.

Results

Preliminary

Mean values on the average 1–7 point scale for each variable were as follows: CBT-competence 5.25 (1.41) and CBT-adherence 4.93 (1.37). The measures of therapist competence and adherence were significantly correlated, r= .692. However, given the theoretical differences between these measures and the importance of understanding their unique contribution to symptom reduction, they were analyzed separately.

Competence and Adherence as Predictors of Outcome

As shown in Table 1, competence did not significantly predict average RCI from baseline to 6 months, but it did significantly predict average RCI from baseline to 12 months and from baseline to 18 months. Specifically, higher competence scores were associated with more negative RCI scores, indicating greater overall change in outcome. Adherence did not significantly predict RCI in any analysis.

Secondary analyses

Baseline measures of symptoms were standardized using Z scores, and an average of the Z scores for all four symptom measures was calculated. This average baseline Z score did not significantly predict competence (β=.105, p=.165), nor did the RCI from baseline to 6 months (β=−.008, p=.921). The average baseline Z score significantly predicted adherence (β=.170, p<.05), but the RCI from baseline to 6 months did not significantly predict adherence (β=−.050 p=.533),

As shown in Table 1, after controlling for symptom change from baseline to 6 months, competence significantly predicted outcomes from 6 to 12 months and from 6 to 18 months. However, adherence did not significantly predict outcome for either of these analyses.

Competence and Adherence Over

Time Average CBT-adherence significantly decreased throughout the course of the study (β = −.283, p<.001). The results for average CBT-competence were in the same direction but only trended towards significance (β = −.136, p=.074).

Predictors of Competence and Adherence

Of the various therapist characteristics, the only significant predictor was years of prior clinical experience, which significantly predicted lower ratings of CBT-competence but not CBT-adherence (see Table 2).

Table 2.

Clinician characteristics as predictors of CBT-competence and CBT-adherence

Measure (Range)	CBT –Competence	CBT-Adherence
	β (R2)	β (R2)
Years of prior clinical experience (2–30)	−.756 (.572)**	−.401 (.160)
Years of formal psychotherapy training (0–3)	.052 (.003)	.169 (.029)
Years of experience conducting psychotherapy (0–17)	−.099 (.010)	.054 (.003)
Years of experience with CBT (0–2)	−.063 (.004)	.097 (.009)
Years of experience with short term dynamic therapy (0–2)	.407 (.165)	.364 (.132)
Years of experience with general supportive therapy (0–8)	.284 (.081)	.200 (.040)
Years of experience with other therapeutic modalities (0–3)	−.011 (.000)	.046 (.002)
Years of experience with anxiety disorders (0–2)	−.168 (.028)	.068 (.005)
Therapist age (27–59)	−.385 (.148)	−.219 (.048)

Discussion

The goal of this study was to investigate the contributions of therapist competence and adherence to the prediction of clinical outcomes of CBT for anxiety disorders delivered by non-expert therapists. A number of previous studies have examined the role of competence and adherence in psychotherapy, but none of them involved non-expert clinicians, the sample most likely to be providing services in community settings such as primary care⁽²⁵⁾. As hypothesized, higher CBT competence predicted better treatment outcomes at 12 and 18 months, although this effect was not present at the 6 month follow-up time-point. On the other hand, CBT-adherence did not significantly predict outcome. These effects remained even after controlling for change in symptoms from baseline to 6 months. One possible explanation is that computer-assisted CBT ensured that therapists met a requisite level of adherence, beyond which additional adherence did not provide further therapeutic benefits. On the other hand, the mean rating of adherence did not reach ceiling levels. Thus, it appears as if the competence with which novice therapists deliver CBT is an important predictor of treatment outcome whereas the degree to which they follow the prescribed session agenda is not.

Adherence significantly decreased over time in the study, and CBT-competence trended in the same direction. It may be expected for adherence to decrease as the time since initial CBT training lengthens, and CBT clinicians sometimes stray from a strict focus on CBT skills and adapt the treatment at times as the relationship with the client develops, and as the complexity of the case deepens⁽⁵⁰⁾. The current sample was very complex, as indicated by their multiple medical and psychological diagnoses, which may have increased therapist drift. However, it is somewhat surprising that competence also trended toward decreasing over time, since prior research indicates that competence improves with more experience and practice^{(12, 28–34)}. On the other hand, there is evidence to suggest that drift in therapist competence is possible within a session, within a study, and within a career⁽⁵¹⁾. In addition, learning any new skill requires enough repetition to influence a change in behavior⁽⁵²⁾, and without continued reinforcement from supervisors, clinicians, and perhaps particularly novice clinicians, may revert to their habitual behavior patterns. Therapists with less prior clinical experience were more easily trained to be competent in delivering CBT than were therapists with more clinical experience. Conceivably, less experienced therapists were more open to training and a new theoretical orientation. We similarly reported that prior clinical experience was negatively associated with clinician performance on CBT training cases⁽³⁶⁾. Providing more clinical examples of how to deliver CBT skills within the computerized program may have increased competence for these clinicians, and future research should address this possibility.

Several potential limitations of this study should be noted. First, the scales used to measure therapist competence and adherence were developed specifically for this study, and although raters demonstrated inter-rater reliability, the lack of psychometric development of the scales is a limitation. Second, this study investigated the use of CBT guided by a computerized program, and therefore it is unclear how well the results would generalize to traditional CBT. Although we controlled for symptom change from baseline to 6 months in the secondary analyses, it is still possible that symptom levels at baseline predict competence. The nature of the data included in this study do not allow for us to control for all temporal confounds of the analyses.. Finally, although the overall sample size of the study was very large (n=1,004), the sample size of study clinicians was necessarily small and therefore this potentially limits the generalizability of these findings to other therapists.

Overall, these findings highlight the important impact that therapist competence has on treatment outcomes in CBT for anxiety disorders. After controlling for baseline differences in patient demographics, symptom severity and number of sessions attended, therapist competence predicted greater symptom reductions and improvements in functioning, while adherence did not. Therapist competence decreased over time, and therapists’ years of prior clinical experience predicted worse therapist competence. The findings highlight the importance of providing sufficient training and supervision to ensure therapist competence. Also, given that competence appears to decline over time, at least in novice clinicians, ongoing therapist training may prove extremely valuable. One option that is cost effective is online training, which has been used effectively to train CBT therapists in rural and remote locations^{(53, 54)}. In addition, it is important to consider the potential interfering effects of prior therapist clinical experiences upon receptivity to training in new approaches. Future research should build on these findings by identifying which training methods best enhance initial therapist learning and maintain competence over time. In addition to continuing to improve the effectiveness of specific therapeutic intervention, improving the effectiveness of the therapists delivering those interventions is an important step towards improving treatment outcomes.

Appendix: Competence and Adherence Measure by Module

CALM Proficiency Evaluation Form

Adherence

All adherence items are indicated by the symbol (A), and were rated on the following Likert scale:

1	2	3	4	5	6	7
Did not			somewhat			fully

Competence

All competence items are indicated by the symbol (C) were rated on the following Likert scale:

1	2	3	4	5	6	7
Did not			somewhat			fully

Getting Started

1. To what extent did ACS explain the purpose of fear and anxiety (A)

2. To what extent did ACS cover “when is anxiety too much?” (A)

3. To what extent did ACS cover “what are anxiety disorders?” (A)

4. To what extent did ACS help the patient identify the anxiety disorder of focus? (C)

Keep Going

1. To what extent did the ACS cover “ongoing practice?” (A)

2. To what extent did the ACS cover planning “long term goals?”(A)

3. To what extent did the ACS cover “How to Maintain Progress?”(A)

4. To what extent did the ACS cover “What are the High Risk Times?”(A)

5. To what extent did the ACS cover “Setbacks?”(A)

6. To what extent did the ACS develop a Practice Plan with the patient? (C)

7. To what extent did the ACS develop a Long Term Goal plan with the patient? (C)

Basics: Education

1. To what extent did the ACS cover the “Causes of Anxiety?” (If ACS skipped this section appropriately, circle N/A) (A)

2. To what extent did ACS cover “What Keeps Anxiety Disorders Going” (i.e., parts of anxiety and anxiety cycle)? (A)

3. To what extent did the ACS cover the “Mechanics of the Body?” (If ACS skipped this section appropriately circle N/A)(A)

4. To what extent did the ACS address whether the patient should review trauma and/or symptom education modules?” (A)

5. To what extent did the ACS cover CALM Tools for Living? (A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did the ACS help the patient identify anxious physical symptoms, thoughts, and behaviors? (C)

8. To what extent did the ACS help the patient complete the Personal Anxiety Cycle? (C)

Basics: Symptom Education

1. To what extent did the ACS cover the “Protective Purpose of the Fight Flight Response?” (A)

2. To what extent did the ACS cover the “Typical Mistaken Beliefs about Physical Symptoms?”(A)

3. To what extent did ACS cover “Summary and Review?”(A)

4. To what extent did the ACS answer patient questions about the physical effects of the fight-flight response? (If ACS skipped this section appropriately, circle N/A) (C)

5. To what extent did the ACS answer patient questions about “Typical Mistaken Beliefs about Physical Symptoms?” (If ACS skipped this section appropriately, circle N/A)(C)

Basics: Trauma Education

1. To what extent did the ACS cover the “Common Effects of Trauma?” (A)

2. To what extent did the ACS cover that the effects of trauma are understandable and not signs of “going crazy.” (A)

3. To what extent did the ACS address whether the patient should review symptom education module?” (A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did the ACS answer patient questions about the symptoms related to trauma? (circle N/A if not applicable.) (C)

Basics: CALM Recording

1. To what extent did the ACS cover the purpose of recording anxiety? (A)

2. To what extent did the ACS cover how to record with the patient?(A)

3. To what extent did the ACS cover what to record with the patient?(A)

4. To what extent did the ACS cover when to record with the patient?(A)

5. To what extent did ACS cover “Summary and Review?”(A)

6. To what extent did ACS help the patient complete a CALM Record form? (C)

7. To what extent did ACS develop a recording schedule with the patient? (C)

Basics: CALM List

1. To what extent did the ACS cover the effect of avoidance?(A)

2. To what extent did the ACS cover how to make a “CALM List?”(A)

3. To what extent did ACS cover “Summary and Review?”(A)

4. To what extent did ACS help the patient identify and rate anxiety provoking situations to develop a CALM List? (the items should be relevant to the targeted disorder and doable). (C)

Calm Breathing 1

1. To what extent did ACS cover “Understanding mechanics of breathing?” (circle N/A If not applicable)(A)

2. To what extent did ACS cover “Understanding what calm breathing is?”(A)

3. To what extent did ACS cover “Learning calm breathing-phase 1(i.e., stomach area, smooth breathing, focused attention)?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS help the patient identify if they are overbreathing? (circle N/A If not applicable)(C)

6. To what extent did ACS deliver the skill of “calm breathing-phase 1(i.e., stomach area, smooth breathing, focused attention)?” (C)

Calm Breathing 2

1. To what extent did ACS cover “Why Calm Breathing is important, again?”(A)

2. To what extent did ACS cover “your practice of calm breathing?”(A)

3. To what extent did ACS cover “Second Phase of Calm Breathing?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS deliver the skill of “calm breathing-phase 2 (i.e., stomach area, smooth breathing, slowed breathing, focused attention)?” (C)

Calm Breathing 3

1. To what extent did ACS cover “Why Calm Breathing is important, again?”(A)

2. To what extent did ACS cover “your practice of calm breathing?”(A)

3. To what extent did ACS cover “Third Phase of Calm Breathing?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS deliver the skill of “calm breathing-phase 3 (i.e., stomach area, smooth breathing, slowed breathing, focused attention while distracting patient)?” (C)

Calm Breathing 4

1. To what extent did ACS cover “Why Calm Breathing is important, again?”(A)

2. To what extent did ACS cover “your practice of calm breathing?”(A)

3. To what extent did ACS cover “Fourth Phase of Calm Breathing?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS deliver the skill of “calm breathing-phase 4 (i.e., stomach area, smooth breathing, slowed breathing, focused attention, and use as a coping skill in moments of anxiety)?” (C)

Calm Thinking 1 (GAD)

1. To what extent did ACS cover “Understanding the anxious mind?”(A)

2. To what extent did ACS cover “what it means to change anxious thinking to calm thinking?”(A)

3. To what extent did ACS cover “Learning about your own anxious mind?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS deliver the skill of “knowing your anxious mind” with an item from the patent’s Calm List? (C)

Calm Thinking 2 (GAD)

1. To what extent did ACS cover “Mistakes in Anxious Thinking?”(A)

2. To what extent did ACS cover “Jumping to Conclusions about Negative Events?”(A)

3. To what extent did ACS cover “Changing Your Anxious Mind: Real Odds?”(A)

4. To what extent did ACS cover “Steps to Calm Thinking: Realistic Odds?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS deliver the skill of “identifying an example of jumping to conclusions? (C)

8. To what extent did ACS deliver the skill of “taking the negative event example and identify why the patient continues to worry about it?”(C)

9. To what extent did ACS deliver the skill of “the real odds for their worry and complete the pie chart?”(C)

Calm Thinking 3 (GAD)

1. To what extent did ACS cover “Review of Jumping to Conclusions and review information as needed?”(A)

2. To what extent did ACS cover “Blowing Things out of Proportion?”(A)

3. To what extent did ACS cover “Facing the Worst and Putting Things Back Into Perspective?”(A)

4. To what extent did ACS cover “Steps to Putting Things Back Into Perspective?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. If the patient did not practice jumping to conclusions with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

8. To what extent did ACS deliver the skill of “identifying an example of blowing things out of proportion (a catastrophic worry)? (C)

9. To what extent did ACS deliver the skill of “assisting the patient in thinking in different ways about their catastrophic worry? (C)

10. To what extent did ACS deliver the skill of “setting up actual copings steps” if the catastrophic worry actually happened?(C)

Calm Thinking 4 (GAD)

1. To what extent did ACS cover “Review of Calm Thinking?”(A)

2. To what extent did ACS cover “Some Tricky Things?”(A)

3. To what extent did ACS cover “Calm Thinking Practice?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. If the patient did not practice putting things into perspective with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

6. To what extent did ACS discuss any tricky issues? (circle N/A if applicable)? (C)

7. To what extent did ACS deliver the skill of helping the patient “practice Calm thinking?” (C)

Calm Thinking 5 (GAD)

1. To what extent did ACS cover “Mistaken Beliefs about Worry?”(A)

2. To what extent did ACS cover “Attempts to Suppress Worry?”(A)

3. To what extent did ACS cover “Ways of Dealing with Beliefs about worry and attempts to suppress worry?”(A)

4. To what extent did ACS cover “Summary and Review” Section?”(A)

5. To what extent did ACS discuss beliefs about worry and attempts to suppress worry with the patient? (circle N/A if applicable) (C)

6. To what extent did ACS develop exercises to help patient see that worry isn’t important and a signal that they’re going crazy (circle N/A if applicable)? (C)

Calm Living 1 (GAD)

1. To what extent did ACS cover “Value of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “Method of Practice with Anxious Situations?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Anxious Situations?”(A)

5. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

6. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

7. To what extent did ACS cover “Summary and Review?”(A)

8. To what extent did ACS deliver the skill of designing a calm living practice with the first item from the patient’s Calm List?(C)

9. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

Calm Living 2 (GAD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List? (C)

9. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

Calm Living 3 (GAD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped) (A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List? (C)

9. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

Calm Living 4 (GAD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Practice with Anxious Situations?”(A)

5. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

6. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

7. To what extent did ACS cover “Summary and Review?”(A)

8. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped) (C)

9. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List?(C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

Calm Feeling 1(GAD)

1. To what extent did ACS cover “Value of Practice with Images Tied to your worries?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “How to Practice with Catastrophic Images?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Catastrophic Images?”(A)

5. To what extent did ACS cover “Summary and Review?”(A)

6. To what extent did ACS deliver the skill of completing imagery descriptions?(C)

7. To what extent did ACS deliver the skill of practicing with images? (C)

8. To what extent did ACS evaluate the imagery practice with the patient? (C)

Calm Feeling 2(GAD)

1. To what extent did ACS cover “Value of Practice with Images Tied to your worries?”(A)

2. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

3. To what extent did ACS cover “Practice with Images?”(A)

4. To what extent did ACS cover “Review of Imagery Practice?”(A)

5. To what extent did ACS cover “Keep Going?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm feeling exercises in the previous week? (Circle N/A if skipped)(C)

8. To what extent did ACS review patient’s imagery practice?(C)

9. To what extent did ACS deliver the skill of practicing with the next highest rated image on the patient’s imagery description list?(C)

10. To what extent did ACS evaluate the imagery practice with the patient?(C)

Calm Thinking 1 (SAD)

1. To what extent did ACS cover “Understanding the anxious mind?”(A)

2. To what extent did ACS cover “what it means to change anxious thinking to calm thinking?”(A)

3. To what extent did ACS cover “Learning about your own anxious mind?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS deliver the skill of “knowing your anxious mind” with an item from the patent’s Calm List?(C)

Calm Thinking 2 (SAD)

1. To what extent did ACS cover “Mistakes in Anxious Thinking?”(A)

2. To what extent did ACS cover “Jumping to Conclusions about Negative Events?”(A)

3. To what extent did ACS cover “Changing Your Anxious Mind: Real Odds?”(A)

4. To what extent did ACS cover “Steps to Calm Thinking: Realistic Odds?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS deliver the skill of “identifying an example of jumping to conclusions (a repeated social worry)?(C)

8. To what extent did ACS deliver the skill of “taking the negative event example and identify why the patient continues to worry about it?” (C)

9. To what extent did ACS deliver the skill of “the real odds for their worry and complete the pie chart?”(C)

Calm Thinking 3 (SAD)

1. To what extent did ACS cover “Review of Jumping to Conclusions and review information as needed?”(A)

2. To what extent did ACS cover “Blowing Things out of Proportion?”(A)

3. To what extent did ACS cover “Facing the Worst and Putting Things Back into Perspective?”(A)

4. To what extent did ACS cover “Steps to Putting Things Back Into Perspective?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. If the patient did not practice jumping to conclusions with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

8. To what extent did ACS deliver the skill of “identifying an example of blowing things out of proportion?” (C)

9. To what extent did ACS deliver the skill of “assisting the patient in thinking in different ways about their catastrophic worry related to social situations? (C)

10. To what extent did ACS deliver the skill of “setting up actual copings steps” if the thing they fear actually happened? (C)

Calm Thinking 4 (SAD)

1. To what extent did ACS cover “Review of Calm Thinking?”(A)

2. To what extent did ACS cover “Some Tricky Things?”(A)

3. To what extent did ACS cover “Calm Thinking Practice?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. If the patient did not practice putting things into perspective with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

6. To what extent did ACS discuss any tricky issues? (circle N/A if applicable)? (C)

7. To what extent did ACS deliver the skill of helping the patient “practice Calm thinking?” (C)

Calm Living 1 (SAD)

1. To what extent did ACS cover “Value of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “Method of Practice with Anxious Situations?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Anxious Situations?”(A)

5. To what extent did ACS cover “Optimizing Direct Experience with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

6. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

7. To what extent did ACS cover “Summary and Review?”(A)

8. To what extent did ACS deliver the skill of designing a calm living practice with the first item from the patient’s Calm List?(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week? (C)

Calm Living 2 (SAD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List? (C)

9. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

Calm Living 3(SAD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List?(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week?(C)

Calm Living 4 (SAD)

1. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “Keep Going?”(A)

3. To what extent did ACS cover “Practice with Anxious Situations (if applicable, including with exaggerated physical symptoms)?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in completing practice with anxious situations and using Calm Thinking strategies so they’ll be more likely to complete the practice next time?” (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List and including deliberately producing exaggerated physical symptoms (if that is appropriate based on calm feeling assessments?)(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week? (C)

Calm Living 5 (SAD)

1. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “Keep Going” (including Practice with Anxious Situations and intentional social mishaps)?”(A)

3. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

4. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

5. To what extent did ACS cover “Summary and Review?”(A)

6. To what extent did ACS help the patient address difficulties in completing practice with anxious situations and using Calm Thinking strategies so they’ll be more likely to complete the practice next time?” (Circle N/A if skipped)(C)

7. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List and including deliberately producing exaggerated physical symptoms (if that is appropriate based on calm feeling assessments?)(Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a social mishap exercise? (C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week?(C)

Calm Feeling 1 (SAD)

1. To what extent did ACS cover “Value of Practice with Physical Symptoms?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “How to Practice with Physical Symptoms?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Physical Symptoms?”(A)

5. To what extent did ACS cover “Summary and Review?”(A)

6. To what extent did ACS deliver the skill of symptom exercise assessment?(C)

7. If appropriate, to what extent did ACS deliver the skill of “other” symptom exercise assessment? (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of practicing with physical symptoms (3x)?(C)

9. To what extent did ACS discuss with the patient how they will conduct their calm feeling exercises over the next week?(C)

Calm Feeling 2 (SAD)

NOTE: IF CALM FEELING 1 (SAD) WAS DISCONTINUED BECAUSE PHYSICAL SYMPTOMS WERE NOT RELEVANT TO PATIENT. DO NOT COMPLETE THIS PEF

1. To what extent did ACS cover “Value of Practice with Physical Symptoms?”(A)

2. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

3. To what extent did ACS cover “Practice with Symptoms?”(A)

4. To what extent did ACS cover “Review of Symptom Practice?”(A)

5. To what extent did ACS cover “Keep Going?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm feeling exercises in the previous week? (Circle N/A if skipped)(C)

8. To what extent did ACS deliver the skill of practicing 3x with the next highest rated exercise on the patient’s practice with symptoms form?(C)

9. To what extent did ACS discuss with the patient how they will conduct their calm feeling exercises over the next week?(C)

Calm Thinking 1 (PD)

1. To what extent did ACS cover “Understanding the anxious mind?”(A)

2. To what extent did ACS cover “what it means to change anxious thinking to calm thinking?”(A)

3. To what extent did ACS cover “Learning about your own anxious mind?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS deliver the skill of “knowing your anxious mind” with an item from the patent’s Calm List? (C)

Calm Thinking 2 (PD)

1. To what extent did ACS cover “Mistakes in Anxious Thinking?”(A)

2. To what extent did ACS cover “Jumping to Conclusions about Negative Events?”(A)

3. To what extent did ACS cover “Changing Your Anxious Mind: Real Odds?”(A)

4. To what extent did ACS cover “Steps to Calm Thinking: Realistic Odds?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS deliver the skill of “identifying an example of jumping to conclusions? (C)

8. To what extent did ACS deliver the skill of “taking the negative event example and identify why the patient continues to worry about it?”(C)

9. To what extent did ACS deliver the skill of “the real odds for their worry and complete the pie chart?”(C)

Calm Thinking 3 (PD)

1. To what extent did ACS cover “Review of Jumping to Conclusions and review information as needed?”(A)

2. To what extent did ACS cover “Blowing Things out of Proportion?”(A)

3. To what extent did ACS cover “Facing the Worst and Putting Things Back into Perspective?”(A)

4. To what extent did ACS cover “Steps to Putting Things Back Into Perspective?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. If the patient did not practice jumping to conclusions with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

8. To what extent did ACS deliver the skill of “identifying an example of blowing things out of proportion”?(C)

9. To what extent did ACS deliver the skill of “assisting the patient in thinking in different ways about their catastrophic worry related to panic attacks? (C)

10. To what extent did ACS deliver the skill of “setting up actual copings steps” if the thing they fear actually happened?

Calm Thinking 4 (PD)

1. To what extent did ACS cover “Review of Calm Thinking?”(A)

2. To what extent did ACS cover “Some Tricky Things?”(A)

3. To what extent did ACS cover “Calm Thinking Practice?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. If the patient did not practice putting things into perspective with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

6. To what extent did ACS discuss any tricky issues? (circle N/A if applicable)? (C)

7. To what extent did ACS deliver the skill of helping the patient “practice Calm thinking?” (C)

Calm Living 1(PD)

1. To what extent did ACS cover “Value of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “Method of Practice with Anxious Situations?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Anxious Situations?”(A)

5. To what extent did ACS cover “Optimizing Direct Experience with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

6. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

7. To what extent did ACS cover “Summary and Review?”(A)

8. To what extent did ACS deliver the skill of designing a calm living practice with the first item from the patient’s Calm List?(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week?(C)

Calm Living 2 (PD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped)(C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List? (C)

9. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

Calm Living 3 (PD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped)(C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List?(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

Calm Living 4(PD)

1. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “Keep Going?”(A)

3. To what extent did ACS cover “Practice with Anxious Situations (including exaggerated physical symptoms)?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in completing practice with anxious situations and using Calm Thinking strategies so they’ll be more likely to complete the practice next time?” (Circle N/A if skipped)(C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List and including deliberately producing exaggerated physical symptoms? (C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

Calm Feeling 1(PD)

1. To what extent did ACS cover “Value of Practice with Physical Symptoms?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “How to Practice with Physical Symptoms?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Physical Symptoms?”(A)

5. To what extent did ACS cover “Summary and Review?”(A)

6. To what extent did ACS deliver the skill of symptom exercise assessment?(C)

7. To what extent did ACS deliver the skill of assessing the patient’s reactions to the symptom exercises?(C)

8. If appropriate, to what extent did ACS deliver the skill of “other” symptom exercise assessment? (Circle N/A if skipped) (C)

9. To what extent did ACS deliver the skill of practicing with physical symptoms (3x)?(C)

10. To what extent did ACS discuss with the patient how they will conduct their calm feeling exercises over the next week? (C)

Calm Feeling 2(PD)

1. To what extent did ACS cover “Value of Practice with Physical Symptoms?”(A)

2. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

3. To what extent did ACS cover “Practice with Symptoms?”(A)

4. To what extent did ACS cover “Review of Symptom Practice?”(A)

5. To what extent did ACS cover “Keep Going?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm feeling exercises in the previous week? (Circle N/A if skipped)(C)

8. To what extent did ACS deliver the skill of practicing 3x with the next highest rated exercise on the patient’s practice with symptoms form?(C)

9. To what extent did ACS discuss with the patient how they will conduct their calm feeling exercises over the next week?(C)

Calm Feeling 3(PD)

1. To what extent did ACS cover “Value of Practice with Physical Symptoms?”(A)

2. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

3. To what extent did ACS cover “Practice with Symptoms?”(A)

4. To what extent did ACS cover “Review of Symptom Practice?”(A)

5. To what extent did ACS cover “Keep Going?”(A)

6. To what extent did ACS cover “Natural Activities?”(A)

7. To what extent did ACS cover “Step Approach For Activities?”(A)

8. To what extent did ACS cover “Optimizing Practice with Activities (Getting the Most from Practice with Anxious Activities)?”(A)

9. To what extent did ACS cover “Final Form: Practice with Activities?”(A)

10. To what extent did ACS cover “Summary and Review?”(A)

11. To what extent did ACS assess the patient’s progress in symptom exercise practice (If patient was complete, the ACS should skip to “natural activities) (C)

12. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm feeling exercises in the previous week? (Circle N/A if skipped) (C)

13. If appropriate, to what extent did ACS deliver the skill of practicing 3x with the next highest rated exercise on the patient’s practice with symptoms form?(C)

14. To what extent did ACS deliver the skill of activities list assessment?(C)

15. To what extent did ACS deliver the skill of designing a practice with the first item from their activities list?(C)

16. To what extent did ACS help the patient use Calm Thinking skills during their Calm Feeling (natural activities) practice?(C)

17. To what extent did ACS discuss with the patient how they will conduct their activities exercises over the next week?(C)

Calm Feeling 4(PD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Practice with Symptoms?”(A)

3. To what extent did ACS cover “Review of Symptom Practice?”(A)

4. To what extent did ACS cover “Keep Going?”(A)

5. To what extent did ACS cover “Natural Activities?”(A)

6. To what extent did ACS cover “Having Trouble Getting Going?” (activities) (Circle N/A if skipped)(A)

7. To what extent did ACS cover “Review of Activity Practice?”(A)

8. To what extent did ACS cover “Keep Going?”(A)

9. To what extent did ACS cover “Optimizing Practice with Activities (Getting the Most from Practice with Anxious Activities)?”(A)

10. To what extent did ACS cover “Final Form: Practice with Activities?”(A)

11. To what extent did ACS cover “Summary and Review?”(A)

12. To what extent did ACS assess the patient’s progress in symptom exercise practice (If patient was complete, the ACS should skip to “natural activities) (C)

13. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm feeling (physical symptom) exercises in the previous week? (Circle N/A if skipped)(C)

14. If appropriate, to what extent did ACS deliver the skill of practicing 3x with the next highest rated exercise on the patient’s practice with symptoms form?(C)

15. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm feeling (activities) exercises in the previous week? (Circle N/A if skipped)(C)

16. To what extent did ACS deliver the skill of designing a practice with the next item from their activities list?(C)

17. To what extent did ACS help the patient use Calm Thinking skills during their Calm Feeling (natural activities) practice?(C)

18. To what extent did ACS discuss with the patient how they will conduct their activities exercises over the next week?(C)

Calm Thinking 1 (PTSD)

1. To what extent did ACS cover “Understanding the anxious mind?”(A)

2. To what extent did ACS cover “what it means to change anxious thinking to calm thinking?”(A)

3. To what extent did ACS cover “Learning about your own anxious mind?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. To what extent did ACS deliver the skill of “knowing your anxious mind” with an item from the patent’s Calm List? (C)

Calm Thinking 2 (PTSD)

1. To what extent did ACS cover “Mistakes in Anxious Thinking?”(A)

2. To what extent did ACS cover “Jumping to Conclusions about Negative Events?”(A)

3. To what extent did ACS cover “Changing Your Anxious Mind: Real Odds?”(A)

4. To what extent did ACS cover “Steps to Calm Thinking: Realistic Odds?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS deliver the skill of “identifying an example of jumping to conclusions? (C)

8. To what extent did ACS deliver the skill of “taking the negative event example and identify why the patient continues to worry about it?”(C)

9. To what extent did ACS deliver the skill of “the real odds for their worry and complete the pie chart?”(C)

Calm Thinking 3 (PTSD)

1. To what extent did ACS cover “Review of Jumping to Conclusions and review information as needed?”(A)

2. To what extent did ACS cover “Blowing Things out of Proportion?”(A)

3. To what extent did ACS cover “Facing the Worst and Putting Things Back Into Perspective?”(A)

4. To what extent did ACS cover “Steps to Putting Things Back Into Perspective?”(A)

5. To what extent did ACS cover “Practice” and instruct the patient to practice this skill for HW with two items from their Calm List?(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. If the patient did not practice jumping to conclusions with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

8. To what extent did ACS deliver the skill of “identifying an example of blowing things out of proportion (a catastrophic worry)? (C)

9. To what extent did ACS deliver the skill of “assisting the patient in thinking in different ways about their catastrophic worry? (C)

10. To what extent did ACS deliver the skill of “setting up actual copings steps” if the catastrophic worry actually happened? (C)

Calm Thinking 4

1. To what extent did ACS cover “Review of Calm Thinking?”(A)

2. To what extent did ACS cover “Some Tricky Things?”(A)

3. To what extent did ACS cover “Calm Thinking Practice?”(A)

4. To what extent did ACS cover “Summary and Review?”(A)

5. If the patient did not practice putting things into perspective with two items from their Calm List, to what extent did ACS deliver that skill? (circle N/A if applicable) (C)

6. To what extent did ACS discuss any tricky issues? (circle N/A if applicable)? (C)

7. To what extent did ACS deliver the skill of helping the patient “practice Calm thinking?” (C)

Calm Living 1(PTSD)

1. To what extent did ACS cover “Value of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “Method of Practice with Anxious Situations?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Anxious Situations?”(A)

5. To what extent did ACS cover “Optimizing Direct Experience with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

6. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

7. To what extent did ACS cover “Summary and Review?”(A)

8. To what extent did ACS deliver the skill of designing a calm living practice with the first item from the patient’s Calm List?(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices?(C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week? (C)

Calm Living 2(PTSD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped) (A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped)(C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List?(C)

9. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

Calm Living 3(PTSD)

1. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

2. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

3. To what extent did ACS cover “Keep Going?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm living exercises in the previous week? (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List?(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice? (C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week?(C)

Calm Living 4(PTSD)

1. To what extent did ACS cover “Review of Practice with Anxious Situations?”(A)

2. To what extent did ACS cover “Keep Going?”(A)

3. To what extent did ACS cover “Practice with Anxious Situations (including exaggerated physical symptoms)?”(A)

4. To what extent did ACS cover “Optimizing Practice with Anxious Situations (Getting the Most from Practice with Anxious Situations)?”(A)

5. To what extent did ACS cover “Final Form: Practice with Anxious Situations?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in completing practice with anxious situations and using Calm Thinking strategies so they’ll be more likely to complete the practice next time?” (Circle N/A if skipped) (C)

8. To what extent did ACS deliver the skill of designing a calm living practice with another item from the patient’s Calm List?(C)

9. To what extent did ACS deliver the skill of “calm behaving” during Calm Living practices? (C)

10. To what extent did ACS help the patient use Calm Thinking skills during their Calm Living practice?(C)

11. To what extent did ACS discuss with the patient how they will conduct their calm living exercises over the next week?(C)

Calm Feeling 1(PTSD)

1. To what extent did ACS cover “Value of Practice with Memories of Your Trauma?”(A)

2. To what extent did ACS cover “What Can I Expect?”(A)

3. To what extent did ACS cover “How to Practice with Trauma Memories?”(A)

4. To what extent did ACS cover “Step Approach to Practice with Trauma Memories?”(A)

5. To what extent did ACS cover “Summary and Review?”(A)

6. To what extent did ACS assess if it is appropriate for the patient to proceed with Calm Feeling exercises for PTSD (If skipped, circle N/A) (C)

7. To what extent did ACS deliver the skill of practicing with memories of trauma starting with the patient’s most anxiety producing memory?(C)

8. To what extent did ACS evaluate the memory practice with the patient?(C)

9. To what extent did ACS discuss with the patient how they will conduct their calm feeling exercises over the next week? (C)

Calm Feeling 2(PTSD)

1. To what extent did ACS cover “Value of Practice with Memories of Your Trauma?”(A)

2. To what extent did ACS cover “Having Trouble Getting Going?” (Circle N/A if skipped)(A)

3. To what extent did ACS cover “Review of Memory Writing Practice?”(A)

4. To what extent did ACS cover “Review of Memory Writing Practice?”(A)

5. To what extent did ACS cover “Keep Going?”(A)

6. To what extent did ACS cover “Summary and Review?”(A)

7. To what extent did ACS help the patient address difficulties in getting going if they did not complete their calm feeling exercises in the previous week? (Circle N/A if skipped) (A)

8. To what extent did ACS review patient’s Memory Writing Practice?(A)

9. To what extent did ACS deliver the skill of practicing writing about the trauma with the same memory as last time or particular parts of that memory that are most anxiety producing?(A)

10. To what extent did ACS evaluate the imagery practice with the patient?(A)

11. To what extent did ACS discuss with the patient how they will conduct their calm feeling exercises over the next week?(A)