“Barriers to Cognitive Behavioral Therapy Homework Completion Scale- Depression Version”: Development and Psychometric Evaluation

Abstract

We conducted a two-phase study to develop and evaluate the psychometric properties of an instrument to identify barriers to Cognitive Behavioral Therapy (CBT) homework completion in a depressed sample. In Phase I, we developed an item pool by interviewing 20 depressed patients and 20 CBT therapists. In Phase II, we created and administered a draft instrument to 56 people with depression. Exploratory Factor Analysis revealed a 2-factor oblique solution of “Patient Factors” and “Therapy/Task Factors.” Internal consistency coefficients ranged from .80 to .95. Temporal stability was demonstrated through Pearson correlations of .72 (for the therapist/task subscale) to .95 (for the patient subscale) over periods of time that ranged from 2 days to 3 weeks. The patient subscale was able to satisfactorily classify patients (75 to 79 %) with low and high adherence at both sessions. Specificity was .66 at both time points. Sensitivity was .80 at sessions B and .77 at session C. There were no consistent predictors of assignment compliance when measured by the Assignment Compliance Rating Scale (Primakoff, Epstein, & Covi, 1986). The Rating Scale and subscale scores did, however, correlate significantly with assignment non-compliance (.32 to .46).

Cognitive Behavioral Therapy (CBT) is an efficacious treatment for depression (Hollon, Shelton, & Davis, 1993; Dobson, 1989; Kazantzis, Deane, & Ronan, 2000; Butler, Chapman, Forman, & Beck, 2006) that is associated with a relatively low risk of relapse post-treatment (Vittengl, Clark, Dunn, & Jarrett, 2007). Yet, the average response rate is no higher than 60 % in controlled studies (Hollon et al., 1992; DeRubeis et al., 2005; Thase et al., 2007; Jarrett et al., 1999). Thus, many patients do not attain full benefit, even when treated by expert therapists following standardized protocols. Among the factors that may influence the probability of response to CBT, homework has long been viewed as a critical element of therapy (Beck, 1979), yet has received relatively little empirical evaluation (Thase & Callan, 2006). Homework, as a predictor of CBT response, was examined in only two randomized trials (Fennell & Teasdale, 1987; Zettle & Hayes, 1987). Neither they, nor other subsequent randomized CBT clinical trials, investigated the barriers to homework adherence.

Improved efforts to systematically examine the barriers to homework adherence, may increase response and remission rates beyond 40 to 60%. The clinical literature offers many recommendations to troubleshoot homework nonadherence (Beck, 1995; Tompkins, 2004; Kazantzis, Deane, Ronan, & L’Abate, 2005; Kazantzis & L’Abate, 2007). We have found, however, that many patients are unable to articulate homework difficulties, and therapists, when faced with fitting all agenda items, are often unable to do a comprehensive assessment. A targeted appraisal via a self-report instrument of common homework barriers may offer an efficient framework for identification and collaborative resolution of non-adherence. Identification of barriers is an essential step in the risk-factor (Kazdin, Kraemer, Kessler, Kupfer, & Offord, 1997) and developmental Intervention process (Hogue, 2002) to decrease undesirable outcomes. Such an instrument may also assist CBT researchers to identify sources of variance in treatment outcome by comparing populations, i.e., those with high or low perceived barriers in relation to homework completion and treatment outcome or as a predictor of relapse/recurrence. Finally, with spiraling health care costs, it is unacceptable to deliver any treatment without all of its active therapeutic components. Systematic investigation of barriers may allow homework, an active component of CBT, to be more fully available to the patient (Hollon et al., 2002). This may ultimately improve the cost-effectiveness of CBT.

At the present time, however, there is no systematic or psychometrically valid instrument to appraise potential patient, therapy, and task factors contributing to homework non-adherence. This paper describes the development and testing of an instrument to assess barriers related to CBT homework completion. It should be noted that this investigation primarily addressed barriers (risk factors to adherence) as opposed to the protective factors that may promote adherence (Kazdin et al., 1997). We chose to focus on barriers as the starting point for the clinical problem solving of non-adherence. It was assumed that protective factors would be operational and already therapeutically maximized when patients were adherent.

We identified barriers in Phase I through interview of 20 depressed CBT patients and 20 CBT therapists followed by a modified qualitative content analysis. We then developed the identified barriers into the items of a draft instrument. In Phase II, we piloted the draft instrument “Barriers to Cognitive Behavioral Therapy Homework Completion Scale- Depression Version,” which will henceforth be referred to as “The Barriers Scale,” in a sample of 56 depressed patients who received 16 to 20 sessions of acute phase CBT, 10 sessions of continuation phase CBT, or a variable amount of CBT sessions in the community. We then examined psychometric properties (construct validity, internal consistency, test-re-test reliability, and predictive validity). We present the results from each phase individually.

Phase I: Scale Development

Method

Participants

We interviewed 20 depressed CBT patients and 20 CBT therapists, recruited in Pittsburgh, PA, for item pool development. We recruited patients and therapists for this ancillary study from a university-based CBT study (Prophylactic Cognitive Therapy for Depression, MH-58356) and additional therapists from the Pittsburgh community.

Criteria for inclusion for outpatients included: DSM-IV diagnosis of Major Depressive Disorder (MDD) at CBT entry; confirmed by SCID-IV interview (First, Spitzer, & Gibbon, 1995); duration of the MDD episode was at least four weeks; age at least eighteen years; and currently in CBT. Patients were excluded for: a diagnosis of bipolar disorder; schizophrenia; obsessive-compulsive disorder; or substance abuse or dependence in the last six months.

Criteria for therapist inclusion was documented training in Beck’s model of CBT or a modified CBT with similar precepts, i.e., Cognitive-Behavioral Analysis System of Psychotherapy (McCullough, 2000). An exclusion criterion was inconsistent use of or omission of homework during therapy sessions.

Measures

We utilized semi-structured interviews comprised of open-ended and clarifying questions with patient and therapists. We included questions for therapists such as “When you’ve been conducting CBT, what seemed to be some of the barriers to the patient completing CBT homework?” JAC asked clarifying questions, such as “What were the patient factors that impacted their ability to complete the homework?” “What particular tasks were more difficult than others for some patients?” We included open-ended questions for patients: “If you’ve ever had difficulty completing your assigned homework in CBT, can you tell me what might have made it difficult for you?” We asked patients to illuminate what the specific problem may have been, i.e., “Could you explain further about how you were feeling that made it hard?” “What was it about the task itself that might have made it too hard for you?”

Procedures

Patients and therapists provided written informed consent prior to study procedures. Basic demographic information was obtained as summarized in Tables 1 and 2. The first author (JAC) interviewed each patient and therapist. She terminated each thirty to sixty minute interview with no new barrier information. The investigators took the following steps to develop the draft instrument:

Table 1.

Descriptive Characteristics of the Patient Samples (Phase I and II)

Variable	Phase I (N = 20)	Phase II (N = 56)
Age (Mean (SD))	49.95 (11.78)	45.84 (12.49)
Gender, n (%) Female	16 (80)	37 (66.1)
Race, n (%) White	19 (95)	49 (88.9)
Marital Status, n (%) currently married	7 (35)	19 (28.6)
Highest Degree Obtained n (%)
High School	6 (30)	20 (35.7)
Greater than High School	14 (70)	36 (64.3)
Years of Education (Mean (SD))	15.50 (2.48)	15.64 (2.71)
Income n (%)
Less than $29,999	8 (40)	19 (33.9)
$30,000 – $49,999	8 (40)	12 (21.4)
Greater than $50,000	4 (20)	9 (16.1)
MDD Age of Onset (Mean (SD))	21.73 (13.71)	24.88 (12.35)
Time in Current Episode (in weeks) (Mean (SD))	26.60 (42.16)	20.26 (26.57)
Time in CBT (in weeks) (Mean (SD))	13.65 (10.74)	8.11 (19.29)
Number of MDD Episode, n (%)
One to Four	6 (30)	24 (42.9)
Greater than Four	13 (65)	31 (53.6)

^**Denotes missing data

Table 2.

Descriptive Characteristics of the Therapist Samples (Phase I and II)

Variable	Phase I N = 20	Phase II N = 13
Age (Mean (SD))	47.35 (9.34)	48.54 (8.67)
Gender, n (%) Female	16 (80)	37 (66.1)
Race, n (%) White	20 (100)	13 (100)
Highest Degree Obtained n (%)
Masters	8 (40)	6 (46.2)
PhD/MD	12 (60)	7(53.8)
Professional Discipline, n (%)
Psychiatrist	1 (5)	N/A
Psychologist	11 (55)	7 (53.8)
Social Work	8 (40)	6 (46.2)
Years of Education (Mean (SD))	21.20 (2.48)	21.08 (2.29)
Training Level, n (%) Advanced	15 (75)	11 (84.6)
Received CBT Supervision, n (%)	19 (95)	13 (100)
Hours of CBT Supervision (Mean (SD))	291.10 (652.44)	262.43 (275.93)
Years Doing CBT Full-Time (Mean (SD))	5.75 (7.27)	7.75 (7.07
Years Doing CBT Part-Time (Mean (SD))	20.00 (6.42)	20.00 (6.42)
Members of Patients Treated with CBT, n (%)
≤ 200	10 (50)	6 (46.2)
≥ 201	10 (50)	7 (53.8)

^**Denotes missing data

Interviews and Initial Identification of Barriers

After reviewing each typed interview, JAC highlighted, numbered, and listed each barrier on the “Barriers Worksheet.” She converted reported barriers, often stated by subjects in sentences, to a simpler descriptor, the barrier keyword, and organized them with an Excel Spreadsheet into “patient,” “therapist/therapy,” and “task” categories. The research assistant, who served as a patient surrogate, conducted an identical procedure. This served as a proxy for a non-professional perspective.

Additional Transcript Review and Barrier Identification by Psychiatric Professionals

Five psychiatric professionals reviewed and identified barriers using the “Barrier Worksheet” from eight randomly selected interviews/per reviewer (four patients, four therapists). JAC converted the barriers to a corresponding barrier keyword. Having additional professional raters discouraged bias in barrier identification.

Identification of Concepts Contained within the “Barriers to CBT Homework” Construct

Three mental health professionals (one psychiatrist, one clinical psychologist, and one master’s prepared nurse) grouped the 283 barrier keywords conceptually, gave them a descriptive conceptual title, and discarded conceptually unrelated keywords. This allowed the identification of the concepts contained within the overarching construct “barriers to homework completion,” i.e., did certain barrier keywords “hang together” consistently within concepts? Understanding the key components within the construct facilitated later selection of representative items for the item pool.

Collapsing Initial Item Groupings from Three Raters

JAC reviewed all conceptual groupings for correspondence. For example, the first rater placed the keyword into his/her conceptual grouping Major Depressive Disorder (MDD) symptoms; the second rater into a conceptual grouping he/she titled “Patient’s emotional state” and the third rater into a conceptual grouping titled “Negative Thought Process/Behavior.” To accommodate these related conceptual themes, JAC then generated an overarching conceptual name “Mood States”. One person (JAC), due to the subjective nature of the conceptual grouping process, conducted this procedure. Quantitative processes or software to collapse three separately named groupings of 283 barriers into one set of concepts was unavailable, necessitating a careful matching of the groupings. Having more than one rater managing this process would have been unwieldy given the many potential combinations. Items not selected by two raters were excluded.

Consistency of Concept Identification/Item Selection

JAC reviewed each of the initial raters’ reviews in relation to whether the barrier keywords comprising a specific concept were identified consistently within an interview, i.e., reliability and the number of times each barrier keyword was identified. We excluded the keyword if not identified in at least two of the raters’ interviews.

We listed the frequency of each keyword in the concepts throughout the 40 interviews, and selected prominent barrier keywords i.e., in a similar fashion to the evident cutoff of a scree plot. The goal was to have prominently reported barrier keywords representing each concept.

Final Scale Development

We conducted several iterations of the barrier keyword conversion questionnaire item wording and then, determined optimal structure, scaling, sentence structure, and instrument format.

Results

The average number of identified barriers per interview were 17 (patients) and 29 (therapists). Through the “item by rater by concept” thematic clustering, the second set of raters identified the following 24 concepts: Dislike/Cynicism Regarding CBT Model; Psychological Readiness; Oppositionality; Therapist Skill; Therapist Qualities; Avoidant Beliefs; Self Efficacy/Self-Esteem; Patient-Therapist Relationship; Patient Background/Demographics; Noncompliance; Mood State; Cognitive Ability/Features; CBT Task Behaviors; Co-morbid diagnosis; Depression Features; External Features; Knowledge CBT Model; Personality Characteristics; Therapist Actions; Procrastination; Nature of Assignment; Positive CBT Tools; Prediction of Good Response; and Patient Beliefs. The raters discarded fifty-three unrelated items. We selected from one to five barrier keywords from each concept for the final item pool. Two criteria determined selection: 1) quantity of barrier keywords per concept and 2) a clear numerical cutoff in the number of interviews in which the barrier keyword was identified. For instance, the concept of CBT Task Behaviors had eighteen barrier keywords. Three foremost items were included in the item pool with frequencies of 41 (Writing Assignments), 37 (Thought Records), and 25 (Homework Connecting Emotions and Thoughts). The next closest was 8 (Alternative Beliefs).

Discussion

Patient and Therapist Sample

Implications for this somewhat non-representative sample relate to content validity. All subjects have recurrent MDD and an average episode length of six months. Although MDD is more likely a chronic/ recurring disorder (Keller, Shapiro, Lavori, & Wolfe, 1982), barriers from an acute MDD profile are excluded. The advanced patient education may be reflected in the types of reported barriers.

Scale Development

To evaluate the process towards a representative item pool, we examined: systematic analysis of the behavioral content domain (Anastasi & Urbina, 1997); items reflect the literature (Fishman & Glaguera, 2003); scale specificity in relation to content domain, setting, and population (DeVellis, 1991) and sampling adequacy reflect the latent variable (DeVellis, 1991).

Developing an item pool for the construct “barriers to CBT homework” was systematic. Following review of the literature, 40 interviews were conducted. A semi-structured interview guide was used with a broad patient/therapist/task theoretical framework (Detweiler & Whisman, 1999). Identification of barriers and the overarching concepts within this construct were made independently by two sets of three reviewers from various psychiatric disciplines. Quantitative procedures, i.e., frequency counts, were brought into play when possible.

The items are reflective of the literature (Beck, 1995; Beck, Rush, Shaw, & Emery, 1979; Bryant, Simons, & Thase, 1999; Detweiler et al., 1999; Davis, 1999; Kazantzis et al., 2005; Tompkins, 2004). Sampling adequacy can be reflected in the nearly 300 reported barriers. While labeling may have resulted in overlap, the resulting number of questionnaire items, and 24 concepts suggest sampling adequacy.

It should be noted that the grouping of barrier keywords into concepts by the second set of 3 raters was, perhaps, the most subjective of the developmental steps. Given the differences in labeling and an inability to “quantify” the process, there may be some bias in the final conceptual groupings. Quantification, i.e., frequencies of identified barriers as a way of determining the foremost items, did assist in deciding the proportion of items for concept representation.

The original scale contained 70 items. In the initial item analysis, five items were dropped due to poor performance, e.g., item-total correlations less than .2. All of the items are negatively phrased, given the focus on barriers. Please see Appendix for revised scale (65-item).

“The Barriers Scale” was developed in a stringent manner and the fund of reported barriers from patients and therapists suggests sampling adequacy. Initial barriers of this scale replicate the theoretical literature. We pilot tested the 70-item draft instrument in a group of 56 depressed patients in CBT to determine the preliminary psychometric properties. It should be noted that the draft questionnaire was developed and tested in a limited homogeneous sample, i.e., depressed patients, so any conclusions that are drawn, refer to this limited population of CBT patients.

Method

Participants

We conducted the study in four academic research programs (Pittsburgh, Dallas, Louisville, and Nashville), several private practices and one mental health outpatient center in Pittsburgh. Eligibility criteria for patients and therapists were identical to Phase I. We recruited a new sample of 56 patients and 13 therapists for Phase II. Study participants provided IRB approved informed written consent prior to data collection. Table 1 and 2 summarizes the characteristics of study subjects in Phase II.

Measures

Barriers to CBT Homework Completion

Barriers to CBT Homework completion was measured by the draft instrument “The Barriers Scale” (Callan, 2007). This self-report Likert-scaled 70-item instrument lists common to homework adherence, e.g., “The therapist gave too much homework.” Items are rated on the degree to which each barrier may have interfered with the completion of the previous session’s homework. Each item was rated on a zero to four scale, zero representing no interference at all and four representing complete interference. Given our desire to have a representative sample of early, mid, and late CBT patients, the scale was given to sequential patients who were in the first eight sessions (first month of treatment), second month of treatment (sessions 9 to 16), or sessions 17 and thereafter (thirteen and thereafter for early responders). We administered the instrument on two consecutive sessions.

Patient adherence to homework assignment

We measured homework adherence through therapist completion of the “Assignment Compliance Rating Scale” (ACRS) (Primakoff et al., 1986; Bryant et al., 1999). It was administered on the two sessions following the administration of “The Barriers Scale.” The ACRS measured the degree to which the patient did the assigned task from the previous session, i.e., quantity, not the quality of the work. ACRS ratings are an operationalized assessment including six categories that range from one (the patient did not attempt the homework assignment) to six (the patient did more of the homework assignment than was requested). For purposes of identifying “good adherence” and partial adherence” the six-point scale was coded as an eight-point scale. The first five items reflect homework that was not done or was only partially done up to 50% completion. The last three items reflect 75% completion, total completion, or more homework done than required. A score of 75 % completion was set as the criteria for good adherence. The only available psychometric data of the ACRS includes correlations between each rater and consensus using the ACRS made by three raters of 57 taped CBT sessions. Correlations between consensus and each rater ranged from .93 to .99 (p< .0001) (Bryant et al., 1999). Percentage of precise agreement ranged from 88.9 % to 97.2% between sets of raters (Bryant et al., 1999).

Depression was measured with the Beck Depression Inventory (BDI)(Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) at the same sessions as “The Barriers Scale.” This self-administered inventory is used to assess the intensity of depression in both depressed and non-psychiatric populations. Twenty-one symptoms and attitudes are rated from 0 to 3 in intensity. In a meta-analysis of 25 studies using the BDI, in both psychiatric and non-psychiatric population, coefficient alpha ranged from .76 to .95 with the mean coefficient alpha of .86 (Beck, Steer, & Garbin, 1988).

Dysfunctional Attitudes were measured by the Dysfunctional Attitudes Scale (Weissman & Beck, 1978)(DAS). The DAS is purported to measure beliefs that represent predispositions to depression, are considered relatively stable (trait beliefs), and reflect negative schemas (Oliver & Baumgart, 1985). The shortened 40-item form of the DAS (Form A) was used in the study and was administered at the same sessions as ”The Barriers Scale” This shortened form has been found to correlate (r= .84) with the DAS-T (total form) and is highly predictive of major depressive disorder (Nelson, Stern, & Cicchetti, 1992).

Demographic Variables included sex, age (in years), educational level (years of education and specific degree obtained), and marital status. Depressive episode history was obtained by providing a description of criteria for Major Depressive Disorder (MDD), defined by the DSM-IV (American Psychiatric Association, 1994) and asking the patient to identify time periods when he/she met those criteria. The Therapist Demographic Questionnaire surveyed variables such as gender, age (in years), educational level (years of education and specific degree obtained), type of training (introductory, intermediate, advanced, extramural), supervision (yes or no), number of years providing CBT, and number of patients treated with CBT.

Procedures

There were three measurement points at consecutive therapy appointments. At Session A, patients and therapists completed the demographic questionnaire, the BDI and the DAS. At Session B this process was repeated without the demographic questionnaire. The patient completed the “The Barriers Scale” and the therapist completed the ACRS. These ratings referenced Session (A) homework. At Session C, the patient completed “The Barriers Scale” and the therapist completed the ACRS. Both referenced Session (B) homework. There was a minimum of two days between therapy sessions and a maximum of three weeks.

Statistical Analyses

An unweighted least squares exploratory factor analysis with an oblique rotation (Promax) was chosen as hypothesized factors were assumed to be correlated. Only factor loadings of .40 or more were considered meaningful and retained. An analysis including all independent items in the scale was unstable, most likely due to the small sample size in relation to the number of items, e.g. the ratio of items to subjects was 1.25 to 1. To reduce the number of items, the concepts derived from the Phase I process were employed as an alternative to the individual instrument items. Scores from each item within a concept were added together to become an item for the factor analysis, i.e., if a concept had 4 items, all of the items were added together to form the “concept as an item.” This aggregated measurement unit is referred to as a “testlet” (Lee, Brennan, & Frisbie, 2000) or “parcel” (Hagtvet & Nasser, 2004); (Kishton & Widaman, 1994); (Little, Cunningham, Shahar, & Widaman, 2002) and (Hall, Snell, & Singer Foust, 1999). Parceling is considered an acceptable option when the parcels are founded on conceptual grounds. It allows for smaller sample sizes because fewer parameters are required for testing. Additionally, compared to item- leveled models, those based on parcels are more parsimonious, allow for fewer residuals to be correlated or items to load on more than one factor, lead to less sampling error, and improve skewed distributions (Hall et al., 1999).

Cronbach’s alpha was used to assess internal consistency reliability. All analyses were tested at the .05 significance level. Stability over time (test-retest reliability) was examined through Pearson Product Moment correlations. Predictive validity was examined using multiple linear regression equations to explore demographic and other independent predictor variables of homework completion. The prediction model included the demographic predictors and the variables of patient depression severity and characteristics, dysfunctional attitudes, therapist training and background, and time in therapy. Divergent validity was examined through Pearson correlations of “The Barriers Scale” and the BDI and DAS, as they represent gold standards in CBT research. Finally, the contrasting groups of patients with “low adherence” and “high adherence” were tested through a binary logistic regression analysis in relation to scale and subscale scores of ”The Barriers Scale” as the independent variables. Predictive validity was also examined with Pearson Product Moment Correlations of “The Barriers Scales” (total and subscale scores) and ACRS. Appropriate transformations and data reflection were used in cases of non-normality.

Results

Factor Analyses

A scree plot revealed a 2-factor solution. The eigenvalues for these factors were 11.61 and 2.7 explaining 48.39% and 11.24% of the variance (total variance was 59.6%). Factor 1 was titled “Patient Factors,” i.e., procrastination, mood state, and patient beliefs and Factor 2 was titled “Therapy/Task Factor,” i.e., therapist skill, therapist qualities, and nature of the assignment. See Table 3 for factor loadings of the concept items.

Table 3.

Barriers to CBT Homework Completion Scale- Depression Version Factor Loadings for Concept Subscales

Factor 1		Factor 2
Concept Name	Loading	Concept Name.	Loading
Opposition	.96	Pt/Therapist Relation	.98
Patient Beliefs	.85	Therapist Skill	.98
Mood State	.83	Therapist Qualities	.95
Predict Good Response	.81	Therapist Actions	.75
Self Efficacy/Self-esteem	.80	Nature of Assignment	.60
Non-compliance	.77	Personality Charac.	.51
Depression Features	.72	Dislike/Cynicism Model	.35**
CBT Task Behaviors	.72
Procrastination	.71
Psychological Readiness	.71
Positive CBT Tools	.71
Pt. Background/Demos	.69
Co-morbidity	.63
Cognitive Abilities	.62
Knowledge CBT Model	.62
External Factors	.58
Avoidant Beliefs	.54

^**It was considered reasonable to include, given the proximity and conceptual meaningfulness.

Internal Consistency

Internal consistency statistics (Session B) are presented in Table 4. The initial item analysis of the 70-item scale revealed 5 items with unsatisfactory item-total correlations (r < .2) that were deleted. Cronbach’s alpha was computed on the 65-item scale with item-total correlations being moderate to high. Item deleted alpha coefficient remained stable for the total scale and the patient subscale. In contrast, coefficient alpha for the therapist/task factors subscale was considerably lower. The possible range for the 24-concept item scale was 0 to 260 points; the patient Subscale ranged from 0 to 180 and the therapy/task subscale ranged from 0 to 80. Scale means indicated positive skewing. Cronbach’s alpha was consistently high with the range of coefficients of .86 to .97.

Table 4.

Barriers to CBT Homework Completion Scale-Depression Version (Measurement B) Internal Consistency

Statistic	Total 70-item	Modified 65-item	Patient Subscale	Ther/Task Subscale*
Cronbach’s Alpha	.97	.97	.94	.86
Item Mean/SD	.65 ± .49	70±48	2.36±1.57	70±.58
Item Mean Range	.02 –1.57	.04+1.57	.50–5.89	1–1.50
Inter-item Correlation
Mean/SD	.29 ±. 21	.3±.18	.53±.12	60±.14
Inter-item
Correlation Range	.24–.83	−.24 –.89	.24±.89	29–.79
Corrected Item Total
Correlation Range	−.08 –.78	.25– .78	.56–.83	.52–.84
Items Below .2	51, 52, 53, 55, 56	none	none	none
Cronbach’s Alpha
if Item Deleted Range	.97–.97	.97–.97	.93–.94	.81–.86
Scale Mean/SD	45.78 ± 37.09	45.03±36.93	40.13± 31.0	4.90± 7.87

^*Denotes internal consistency done on 24-item concept scale

Temporal Stability

All correlations for the test-retest analyses were significant, the entire scale (r = .95) and the Patient subscale (r = .94) had a much more robust association, indicating greater stability over time as compared to the therapy/task subscale (r = .72). The lower correlation of the therapy/task subscale highlighted potential differences in performance.

Prediction of Homework Adherence

Five of six correlations examining the association with “The Barriers Scale” and subscale scores and ACRS Scores ranged from .32 to .46 (See Table 5), indicating a moderate relationship. The model of predicting adherence group membership with the Patient subscale through binary logistic regression correctly identified nearly 80% and 77% at both measurement points. Low patient subscale scores predicted high adherence. At a cut point of .5, the sensitivity was .80 and .77, respectively, at session B and C. The specificity was .66 at both time points.

Table 5.

Pearson Product Moment Correlations of Barriers to CBT Homework Completion Scale Scores and Assignment Compliance Rating Scale Scores

Variable	Total Scale	Patient Subscale	Therapist/Task Subscale
Correlation at B	.32*	.33*	.18
Correlation at C	.46*	.46*	.34*

^*p ≤ .05

Demographic, patient’s clinical attributes and therapist covariates were used to determine prediction of assignment compliance at both time points. This complete regression model containing all of the variables of interest as well as the barriers scores did not predict homework adherence (Adjusted R² of −0.08 at measurement B and 0.10 at measurement C).

Divergent Validity

Pearson Product Moment correlation data examining the relationship between the BDI and DAS to “The Barriers Scale” were significant for the BDI at session B (Entire scale and the patient subscale) and the DAS for sessions B and C. The significant correlations are low to moderate (.25 to .41 for BDI only at session B and .26 to .34 for the DAS at sessions B and C).

Discussion

This scale demonstrated high internal consistency in the entire scale and the subscales. The total scale and the Patient Subscale demonstrated high levels of stability over time while the therapist/task subscale, while adequate, was less robust. Factor analysis of the original scale proved untenable. An alternate strategy using the composite scores from the concepts produced a 2-factor solution which explained 60% of the variance. The Pearson correlations demonstrated a consistent moderate relationship between the Patient Subscale, the entire scale, and the therapist/task subscale (on one occasion) to homework non-completion as measured by the ACRS, suggesting reasonable predictive validity. The logistic regressions indicate that the Patient subscale is able to identify those that are likely to be good adherers (> 75% adherence). The Therapist/Task Subscale did not evidence this capability.

Conclusions that are drawn are based on a primarily white, middle-aged, educated sample with recurrent MDD from academic settings. The therapist sample is also similar to that of Phase I but for a greater proportion of academic therapists. Providing CBT in a prototypical manner may have reduced the scores on the Barriers scale and altered the outcome measure of homework completion as evidenced by the positive skewing in the scale scores and negative skewing in the ACRS. Specifically, the Cognitive Therapy Rating Scale (CTS) scores of this group evidenced a high level of CBT expertise. The mean CTS rating score was 45.6 + 6.1; the median was 45.0 and the range was 24 to 65 across 15 therapists and 377 ratings. Protocol eligibility standards require a CTS of 40. Thus, it is likely that this highly skilled group of therapists employed routine tools to reduce barriers and encourage adherence to homework. Findings regarding therapist subscales may be reflective of the therapist skill level in the sample. Additionally, the small sample of 56 patients, suggests that additional evaluation of this instrument is called for. A sample of 5 to 10 subjects per item (Nunnally, 1978) will be targeted for future studies.

In self-report identifying barriers related to homework completion, it is conceivable that patients did not score items related to perceived therapist deficiencies. Patients were reassured of strict confidentiality; social desirability may have impacted ratings about barriers related to therapist performance. Nevertheless, this may provide some guidance that, self-report, as a vehicle to report therapist barriers, may not be practical. It is possible, however, that patient factors are the most central in understanding barriers to CBT homework adherence. In subsequent analyses, it can be seen that the Patient subscale was preeminent in relation to prediction of homework adherence. This should be considered a tentative conclusion, given the homogeneous and limited patient and therapist sample. Factors related to the therapist, it appears, have limited predictive power in relation to barriers scores and noncompliance to homework. Again, this may be related to the high level of therapist skill in this study.

Seventy-four percent of the subjects met the criteria for good adherence. This is on the higher range of the average across clinical populations of an average of 50 % adherence (Dunbar-Jacob, 2000). The above analyses indicate the instruments ability to identify good and bad adherer’s even when there are so few poor adherers and to demonstrate a significant moderate relationship between the instrument and the outcome measure.

The two-factor solution of Patient Factors and Therapist/Task Factors corresponds to some degree to the initial Detweiler & Whisman model of Patient, Task, and Therapist factors affecting homework adherence. Our findings concur with Helbig & Fehm (Helbig & Fehm, 2004),that patient factors were the most robust in predicting adherence.

Finally, the significant low to moderate correlations with the BDI and the Total and Patient Subscale at measurement B and the DAS with all of the scales and subscales at measurement C, indicate related but not redundant concepts. Factors related to depressed mood and dysfunctional attitudes most likely inter-relate with perceived barriers to homework completion.

Conclusions

We set out to create an instrument that could be utilized by clinicians, patients and researchers to identify common barriers to the completion of CBT homework assignments in a depressed population. In Phase I we identified 70 items that represented common barriers to CBT homework completion for the draft instrument. In Phase II we tested the draft instrument in a sample of 56 depressed CBT patients. The findings from this study suggest the instrument “The Barriers Scale” has encouraging preliminary psychometric properties and provides a tool that uniquely assesses this construct.

Limitations to the present study primarily relate to use of a small homogeneous convenience sample of depressed patients as opposed to a broader range of people with more diverse demographic characteristics seeking therapy for a broader range of conditions. The sample size was insufficient to suitably conduct a factor analysis (using the items) and to test a complicated prediction model. Given this, findings related to factor structure and prediction, based upon the regression models and correlations, should be considered tentative. This instrument, nevertheless, addresses a critical obstacle in the execution of CBT therapy. homework is a central agenda item. This tool offers a systematic method with encouraging psychometric properties, to explore the predominant barriers. Additionally, it may suggest a course correction on the assignment of homework and the management of the therapy, i.e. too many assignments, homework too difficult, therapist not checking, etc. The instrument takes approximately five to ten minutes to complete so frequent usage is not untenable. It becomes, then, a prescriptive tool to be examined in a collaborative manner to advance patient goals. Additionally, use of the tool on a routine basis strengthens the significance of homework in the CBT model and, thereby, potential improvement of clinical status.

Additional research is required with a larger more representative sample to evaluate the factor structure and examine the regression model with greater power to detect an effect. We intend to reduce the length of the scale while still maintaining the reliability and proportional concept representation. At this stage of scale development, over-inclusiveness and some redundancy are considered desirable, so a 65-item instrument is reasonable (DeVellis, 1991). Regular use dictates an even user-friendlier version. Thus, the potential future directions may also include modifying the instrument to be completed in five minutes or less and examining the utility of this instrument in other populations.