Who is at risk for dropout from group cognitive-behavior therapy for insomnia?

Abstract

Objective

The aim of the present study was to identify characteristics of patients who are at risk for dropout from a 7-session group cognitive-behavior therapy for insomnia (CBT-I) in a clinic setting using the receiver operating characteristics curve (ROC) approach.

Methods

Two separate ROC analyses were conducted using predictor variables taken from questionnaire packets and sleep diaries collected at baseline including age, gender, Beck Depression Inventory (BDI), Morningness-Eveningness Questionnaire, Beliefs and Attitudes about Sleep, use of sleep medication, sleep onset latency (SOL), wake time after sleep onset (WASO), and total sleep time (TST).

Results

The first ROC analysis was conducted on the entire sample of 528 patients with treatment completion versus dropout (non-completion) as the outcome variable. No significant predictor variables were found in this analysis. The second ROC analysis was conducted on the 211 patients who did not complete treatment with early termination (prior to fourth session) versus late termination (at or after fourth session) as the outcome variable. The results revealed that patients who reported an average baseline TST < 3.65 hours were at greatest risk for early termination. Sixty percent of patients in this group terminated early compared to 9.3% of patients with TST ≥ 3.65 hours. Among patients with TST ≥ 3.65 hours, 22% of those with BDI scores ≥ 16 were early dropouts compared to 4.3% of those who reported BDI < 16.

Conclusion

These findings indicate that short sleep duration and elevated symptoms of depression at baseline are associated with increased risk of early termination from CBT-I.

INTRODUCTION

Chronic insomnia is a prevalent problem with approximately 33% of American adults reporting at least one nighttime symptom of insomnia occurring every night or nearly every night during the past year (1). Cognitive-behavior therapy for insomnia (CBT-I), a multi-component treatment package that includes stimulus control, sleep restriction, relaxation exercises, and cognitive restructuring techniques, has demonstrated efficacy for the treatment of insomnia (e.g., (2–5)). Meta-analyses have reported large effect sizes for reducing sleep onset latency and improving sleep quality and medium effect sizes for reducing wake time after sleep onset and increasing total sleep time (6–9). In addition, treatment outcome studies have found that CBT-I is superior to pharmacological treatment in maintaining these benefits beyond the termination of treatment (3–5). The accumulated evidence has led the American Academy of Sleep Medicine to recommend CBT-I as a standard treatment for chronic insomnia (10).

Despite the strong evidence supporting the efficacy of CBT-I, little is known about factors related to attrition during treatment. Early treatment outcome studies were inconsistent in reporting attrition, thus making it difficult to estimate the rate of dropouts. The recommendations outlined in the CONSORT statement (11) have improved the reporting of patient flow in randomized clinical trials (RCTs) and more recent RCTs using individual or group CBT-I for primary insomnia have revealed very low rates of dropout for participants in the CBT-I condition, ranging from 0% to 8% (2–5). However, RCTs typically recruit homogenous samples, most commonly limited to patients with primary insomnia. These studies employ rigorous protocols that closely monitor attendance and use other active strategies to minimize attrition, thus rendering it difficult to generalize these findings to clinical settings, which often do not allocate resources for patient retention.

Indeed, studies conducted in clinical settings have reported much higher dropout rates. Two effectiveness studies examining a six-session group CBT-I in clinical settings reported noncompletion rates of 13.7% (12) and 34.0% (13) in the CBT-I groups. Studies using a case replication series of patients presenting to sleep clinics for CBT-I have reported a wide range of non-completion rates, from 9.7% to 38.8% (14–18). One factor that that has been problematic in identifying dropout rates has been the inconsistency in operationalizing the term dropout. Some studies have reported dropout rates based on early termination, or those who dropout prior to receiving an adequate dose of treatment. When a minimum adequate dose of treatment was defined as attending at least four sessions of CBT-I, early termination rates of 9.7% have been reported for group CBT-I (17) and 30.3% to 38.8% for individual CBT-I (14, 18). In contrast, other studies have reported dropout rates based on non-completion, or the failure to complete the treatment protocol regardless of when termination occurred. These studies have reported noncompletion rates of 13.7% to 34.0% for group CBT-I (12, 13, 16) and 30.0% for individual CBT-I (15). Clearly, the dropout rate is considerably higher in clinic settings compared to RCTs and the rate varies depending on the timing of termination from CBT-I. Therefore, identifying characteristics of clinic patients who dropout of treatment, especially early dropouts, could lead to improvements in the care these patients receive.

Currently, very little is known about risk factors that predict patient dropout. Some evidence suggests that the severity of sleep disturbance is related to withdrawal from treatment but the findings are inconsistent. Perlis and colleagues (18) found that patients who terminated from treatment prior to the fourth session reported a greater number of awakenings and less total sleep time at baseline relative to those who completed treatment. In contrast, Morgan et al. (13) found that non-completers reported less severe symptoms of sleep disturbance at baseline, as measured by the Pittsburgh Sleep Quality Index (PSQI), compared to treatment completers. Treatment process variables have also been examined in the context of attrition. Pre-treatment ratings of the acceptability of psychological treatment have been reported to be lower among non-completers compared to completers of behavioral treatment (16). A recent study examining therapeutic elements in group CBT-I found that patients who dropped out prior to the fourth session perceived their therapist as more critically confrontive during the first session compared to patients who continued in treatment (19). Although these studies provide preliminary insights, further research examining predictors of dropout among a set of clinically-relevant variables would be particularly useful.

The receiver operating characteristics curve (ROC) is one analytic strategy that is well-suited for identifying predictors or characteristics of those who are at risk for a particular dichotomous outcome, such as dropout from treatment. ROC utilizes a signal detection technique that can be used to evaluate the sensitivity and specificity of medical tests or to simultaneously evaluate a set of variables for the prediction of a binary outcome (20). Unlike linear models that require a priori entry of interactions to detect these effects, ROC can provide information about interactions among variables using an ROC tree that identifies sub-groups based on cut-off scores. This hypothesis generating technique has been used in other naturalistic studies using large samples with a specific set of predictor variables (e.g., (21–24)). Using the ROC approach to identify predictors of dropout from CBT-I may yield important clinical information about patients who do not stay in treatment.

The aim of the present study was to identify characteristics of patients who are at risk for dropout from group CBT-I using a set of variables collected in a clinic setting. The ROC approach was employed because of its suitability for conducting exploratory analyses on a number of predictors for a binary outcome such as completion versus dropout from treatment. The set of predictor variables selected in this study are clinically relevant and commonly collected during the course of behavioral treatments for insomnia, among which several measures were recommended as standard measures for the assessment of insomnia in research settings (25). To address previous discrepancies between non-completers and early dropouts, two separate ROC analyses were conducted. First, the ROC analysis was conducted on the entire sample to examine predictors of completers versus dropouts of treatment. A second ROC analysis was conducted on the subgroup of non-completers to examine predictors of early versus late dropouts. By exploring data collected from clinic patients, these analyses serve to generate hypotheses that can lead to further investigations aimed at improving the delivery of CBT-I.

METHOD

Participants

The present study was conducted using archival data collected from a series of 528 patients who attended a CBT-I group program between March 1999 and May 2004 at the Stanford Sleep Disorders Clinic. The average age of patients was 47.9 years (SD = 14.3 years) and 57.8% of the sample were female. The study was approved by the Institutional Review Board (IRB) at Stanford University Medical Center and all participants provided written informed consent or were exempt by a waiver obtained from the IRB.

Treatment

The CBT-I group program is an outpatient service offered at the Stanford Sleep Disorders Clinic. All clinic patients first received an initial evaluation by a sleep specialist and were subsequently referred to the CBT-I program if the evaluation revealed evidence of clinically significant symptoms of insomnia and the sleep specialist determined that the patient could benefit from CBT-I. Referred patients included those with co-existing sleep disorders (e.g., sleep apnea, restless legs syndrome, periodic limb movement disorder), psychiatric disorders (e.g., depression, anxiety), medical conditions, as well as those on hypnotic medications.

The group CBT-I program consisted of a multi-component treatment delivered in seven, 90-minute sessions over a nine week period with typical group sizes between 8–15 patients (26). All groups were led by a licensed psychologist who was certified in 2003 as a behavioral sleep medicine specialist (C.B.S.M.). The first five sessions were conducted weekly and the final two sessions were conducted bi-weekly. During the first session, the therapist provided an overview of the treatment program and obtained relevant information from each group member regarding his or her insomnia symptoms (e.g., sleep patterns, medication use, co-existing medical conditions). The second session consisted of education about normal sleep and factors that contribute to its disturbance. The third session consisted of a discussion on the relationship between sleep and cognitive, emotional, and physiological arousal and patients were taught specific strategies to reduce arousal (e.g., deep breathing, mental imagery). The fourth session included instructions for stimulus control and sleep restriction, with the therapist prescribing an individually tailored sleep program for each patient incorporating these instructions. During the remaining sessions, the therapist reviewed each patient’s progress and recommended adjustments to the behavioral program as necessary and cognitive restructuring when indicated.

Measures

Sleep Diaries

Patients completed prospective sleep diaries every morning for one week at baseline before receiving any specific treatment recommendations. In the present study, several variables were derived from the sleep diaries. First, the items for sleep onset latency (SOL), wake time after sleep onset (WASO), and total sleep time (TST) were averaged across the week (minimum 6 nights of data) as a measure of self-reported sleep at baseline. Second, the use of any medication to facilitate sleep on any night was coded as a dichotomous variable so that patients were categorized as a user or non-user of sleep medication at baseline. Sleep diaries are routinely used for clinical and research purposes and are considered the standard of practice for measuring sleep in insomnia populations (27).

Dysfunctional Beliefs and Attitudes about Sleep (DBAS)

The DBAS was originally developed as a 30-item scale by Morin and colleagues (28). In a subsequent study, Espie and colleagues (29) found that a 10-item version (DBAS-10) was highly correlated with the original 30-item DBAS (r = .83) and has satisfactory internal consistency (Coefficient alpha = .69) (29). In the present study, the total score from the DBAS-10 was used as a measure of baseline beliefs and attitudes about sleep.

Morningness-Eveningness Composite Scale (MECS)

The MECS (30) is a 13-item scale used to determine an individual’s preference for various activities and ease of rising in the morning. The scale includes nine items from the Horne-Östberg Mornigness-Eveningness scale (31) and four items from the Torsvall and Ǻkerstedt scales (32). The MECS has excellent internal consistency (alpha = .87) and demonstrated psychometric properties that are comparable or better than the Horne-Östberg and Torsvall and Ǻkerstedt scales (30). The MECS was used as a measure of circadian preference at baseline and was included in the present analyses given recent findings that insomnia patients who endorse an evening chronotype report sleep/wake irregularities and waking distress that is above and beyond the level of insomnia severity (33).

Beck Depression Inventory (BDI)

The BDI is a 21-item self-report scale used to assess symptoms of depression (34). The scale has high internal consistency (Coefficient alpha = .87), strong evidence of validity, and appears to have one underlying factor (35). In the present study, the total score on the BDI was used as a measure of depression symptom severity at baseline.

Data Analysis

The receiver operating characteristic curve (ROC) analysis using the ROC4 program (found at http://mirecc.stanford.edu and described in (20)) was conducted on a set of variables collected at baseline to identify predictors of dropout from CBT-I. ROC is a non-parametric technique that is capable of evaluating multiple potential predictors without making restrictive assumptions (e.g., linearity, additivity, homoscedasticity) that is required of linear models. Also, a unique feature of the ROC program is that it allows the user to designate the criterion for identifying the best variable by adjusting the weight in kappa in order to optimize sensitivity (i.e., emphasis placed on avoiding false negatives), specificity (i.e., emphasis placed on avoiding false positives), or efficiency (i.e., equal emphasis placed on both types of errors). The decision to adjust the weighted kappa is based on clinical importance of false negatives versus false positives. For each independent variable (IV), the program searches for a cut-point that optimizes the balance between sensitivity and specificity for predicting the outcome of interest (e.g., dropout). Once the best predictor (and optimum cut-point) is identified, the group with the success criterion is tested against a stopping rule (cut-point significant at p < .01 level). If it fails the stopping rule, no further action is taken. If the group passes the rule, the sample is divided into two sub-groups based on the predictor variable. The analyses are then restarted for each of the two subgroups in an iterative process until the stopping rule is encountered (either a subgroup reaches a sample size of n < 10 or the optimal test is not statistically significant at the .01 level).

In the present study, two separate analyses were conducted. First, the ROC analysis was conducted on the entire sample (n = 528) with completion (attended the last session) versus dropout (failed to attend the last session) as the outcome measure. Subsequently, a second ROC analysis was conducted on the subset of patients (n = 211) who did not complete treatment with early termination (dropout prior to fourth session) versus late termination (dropout at or after fourth session) as the outcome measure. This criterion was selected because patients who terminate prior to the fourth session have yet to receive the core behavioral components (i.e., sleep restriction and stimulus control) that have demonstrated efficacy in the treatment of insomnia and are often considered to be the most important components in CBT-I programs. In addition, this criterion has been used previously as a cut off for the “minimum adequate dose” in case replication series studies (14, 17, 18). The set of predictors (IVs) in the present study include: age at the beginning of treatment, gender, BDI, MECS, DBAS, sleep medication (user or non-user), SOL, WASO, and TST. These IVs were selected because they are commonly collected in CBT-I programs and provide clinically relevant information on sleep parameters, sleep-related cognitions, circadian preferences, depression symptoms, and sleep medication use. In the ROC analysis, the weight for kappa was set at .50 so that false negatives and false positives are given equal consideration.

RESULTS

Termination rates

For the total sample of 528 patients, the average session of termination (i.e., last session attended) was 6.13 (SD = 1.39), with 317 out of the 528 patients (60%) completing the 7-session treatment. Of the 211 non-completers, 35 patients terminated prior to the fourth session and were considered early drop-outs while 176 patients terminated between the fourth and sixth sessions and were considered late drop-outs. Please see Table 1 for a summary of the data on termination by session.

Table 1.

Termination by Session.

Last Session Attended	n	Percent of Sample
1	11	2.1
2	5	0.9
3	19	3.6
4	32	6.1
5	54	10.2
6	90	17.0
7	317	60.0
Total	528	100.0

Completion versus dropout

Results from the first ROC analysis yielded no significant predictor variables when completion versus dropout was used as the outcome measure on the entire sample. The best predictor of dropout was SOL with an optimal cut-point of ≥ 0.34 hours (χ² = 5.35, p > .01) but this variable did not pass the stopping rule. Table 2 presents descriptive data for completers.

Table 2.

Subgroup characteristics at baseline

Variable	Completers		Non-Completers
			Early Dropouts		Late Dropouts
	M	SD	M	SD	M	SD
Age	48.65	14.36	47.73	16.11	46.61	13.77
Gender (% female)	55.8%		71.4%		58.5%
BDI	12.60	7.65	16.70	11.92	13.57	8.32
MECS	35.99	8.97	33.43	10.04	35.14	9.42
DBAS	61.81	15.31	63.94	15.70	62.52	14.25
Sleep Med (% yes)	58.0%		54.3%		55.7%
SOL (hours)	0.59	0.53	0.80	0.73	0.62	0.51
WASO (hours)	1.50	0.93	1.53	1.18	1.34	0.84
TST (hours)	6.18	1.26	4.87	1.76	6.24	1.36

Note. BDI = Beck Depression Inventory; MECS = Morningness-Eveningness Scale; DBAS = Dysfunctional Beliefs and Attitudes about Sleep; Sleep Med = use of any sleep medication at baseline; SOL = Sleep Onset Latency; WASO = Wake Time After Sleep Onset; TST = Total Sleep Time.

Early versus late dropout

A second ROC analysis conducted on the subset of dropouts (n = 211) revealed two predictor variables of early dropouts (see Figure 1 for ROC tree). At the first level, the best predictor variable was TST with an optimal cut-point of < 3.65 hours (χ² = 21.70, p < .001). Of the 10 patients who reported TST < 3.65 hours, 60% were early dropouts. For this group, the stopping rule went into effect and the ROC analysis did not further differentiate any sub-groups. In contrast, out of the 140 patients who reported TST ≥ 3.65 hours, 9.3% were early dropouts. Subsequently, this group was further differentiated by BDI score, with a cut-point of BDI ≥ 16 (χ² = 10.27, p < .01). Of the 41 patients who reported TST ≥ 3.65 hours and BDI ≥ 16, 22% were early dropouts. In contrast, out of the 94 patients who reported TST ≥ 3.65 hours and BDI < 16, only 4.3% were early dropouts. At this point, the stopping rule went into effect for all groups. No other significant variables were found. Table 2 presents descriptive data for early and late dropouts.

Figure 1.

ROC Tree of subgroups with different risk for early dropout. ED = Early Dropout. The number of subjects who were early dropouts are in parentheses.

DISCUSSION

The goal of this study was to identify characteristics of patients who are at risk for dropout from a CBT-I group program. Overall, the findings revealed that self-reported TST and depressive symptoms are two significant predictor variables of early dropout while no significant predictor variables for non-completion of treatment were found in this clinic sample. Specifically, the ROC analysis revealed that patients who reported an average of less than 3.65 hours of TST at baseline were most likely to dropout early. Among patients who reported at least 3.65 hours of TST, those who also reported BDI scores at or above 16 were most likely to dropout early. These findings provide evidence of the impact of short sleep duration and elevated symptoms of depression on patient attrition in group CBT-I along with preliminary recommendations for cut-off scores on these variables.

This study was unique in that separate ROC analyses were conducted to examine predictors for treatment completers versus dropouts and subsequently for early versus late dropouts among the subgroup of non-completers. Interestingly, no significant predictor variables were found in the ROC analysis on treatment completion. Previous studies have found that greater severity of sleep disturbance and higher treatment acceptability ratings were related to completion of treatment (13, 16). In this study, the absence of predictors of non-completion might be related to the limited set of variables that were collected and a larger or different set of predictors might have yielded different results. Unfortunately, variables such as the reason for termination and sleep parameters at the time of termination in this study were not routinely collected. Thus, it is unclear if the non-completers terminated because they had achieved early improvements in their sleep, if they were unsatisfied with treatment, or if they developed complications with treatment. Given recent evidence that an optimal dose of individual CBT-I is four sessions (36), it is possible that some patients who terminated after four group sessions had already received sufficient therapeutic gains from treatment. Therefore, future research should include an expanded set of variables, including patient expectations, therapeutic alliance, and pattern of medication use (e.g., dose and timing), to provide a more comprehensive evaluation of the potential risk factors for dropout.

The ROC analysis on early versus late dropouts revealed that short sleep duration was the greatest risk factor for early dropout, a finding consistent with a previous report from Perlis and colleagues (18). In that study, early dropouts of individual CBT-I reported an average TST of 4.08 hours and those who received more than four sessions reported an average TST of 4.87 hours. In the present study, the ROC analysis identified a cut-point of TST < 3.65 hours as optimizing the balance between sensitivity and specificity for predicting early termination in group CBT-I. Taken together, the emerging evidence indicates that patients who report an average TST approximately four hours or less appear to be at high risk for early termination from both individual and group CBT-I. Further research should investigate this finding as an a priori hypothesis and examine the reasons for dropout in this high risk group. The ROC analysis also revealed important information regarding the interaction between TST and BDI scores as indicated by the branching of the sub-groups. Specifically, patients who reported TST ≥ 3.65 hours and BDI ≥ 16 are at significantly greater risk for early dropout compared to patients who reported TST ≥ 3.65 hours and BDI < 16. Although previous research has found that individuals with insomnia and elevated depression symptoms can benefit from CBT-I (37–39), it is unclear how depression might impact adherence to treatment. The present findings indicate that the interaction between total sleep time and symptoms of depression should be further examined. Future studies using linear models to investigate treatment dropout should consider a priori specification of these interactions in the model for hypothesis testing.

Another strength of this study was the large sample size, which provided a more stable estimate of dropout rates from group CBT-I in a clinic setting than can be obtained from smaller samples. We found that 40% of the clinic sample did not complete the full course of treatment, with nearly 7% of patients terminating prior to the fourth session. The non-completer rate was slightly higher than reported by other case replication series studies (14–16, 18) and the early dropout rate was similar to that reported when CBT-I was delivered in groups (17) and lower than that reported when treatment was delivered individually (14, 18). Collectively, these findings suggest that roughly 14% to 40% of patients in clinic settings fail to complete the full course of CBT-I delivered individually or in groups, with early dropout rates between 7% to 10% for group CBT-I.

The present study was designed to answer the question of who is at risk for dropout from group CBT-I but further research is needed to address the question of why these individuals are at risk. One possibility is that patients who terminate early are not appropriate candidates for group CBT-I. In the present study, patients were referred to the CBT-I group program based on the referring physician’s clinical judgment that the patient would benefit from the program, but no formal screening or treatment algorithm was employed, as such guidelines are incomplete and not uniformly adopted. Clinical guidelines based on conceptual models (40) have recommended a threshold of SOL, WASO, or early morning awakening (EMA) of 30 minutes or greater as a minimum threshold to be an appropriate candidate for CBT-I, but no recommendations were made regarding an upper limit of insomnia severity as a counter-indication of CBT-I. The present findings indicate that further investigation of this issue is warranted. A second possibility is that patients who are at risk for early termination can benefit from CBT-I but may require additional clinical attention to stay in treatment. Patients who have substantial sleep deprivation (i.e., TST < 3.65 hours) might not be willing to endure the process of CBT-I, which typically takes a few weeks for increased total sleep time to emerge. For these patients, the use of adjunctive strategies such as an early trial of hypnotic medications to increase total sleep time or the use of a specific technique, such as motivational interviewing during the diagnostic interview might improve retention for these patients. Further testing of this hypothesis could improve the overall treatment of insomnia by providing clinicians with better guidance to match treatment strategies with patient characteristics.

Given that the present study was conducted to explore data provided by clinic patients, these conclusions should be considered within certain limitations. First, it is important to emphasize that the ROC analysis is a method for generating hypotheses rather than testing hypotheses. Therefore, replication of these findings should be tested in other samples before the predictor variables and cut-points identified in this study can be adopted as recommendations for clinical decision making. At present our findings should be viewed as providing preliminary guidelines for identifying patients at risk for dropout from group CBT-I. As mentioned earlier, information on the reason for dropout was not collected in this study. Although this information is difficult to obtain in routine clinical practice, future research should consider interviewing patients who dropout or having them complete questionnaires about treatment satisfaction as part of routine clinical care. Finally, it should be noted that the present analyses were conducted on CBT-I delivered in a 7-session group format. It remains unclear if these findings would generalize to the delivery of CBT-I using other modalities (e.g., individual psychotherapy, minimum contact therapy) or different protocols (e.g., fewer sessions, or different order of treatment components). Despite these limitations, the present findings provide important indications of the clinical relevance of short sleep duration and elevations in depressive symptoms upon presentation to a CBT-I program. It is hoped that these findings will stimulate further investigation into why these risk factors are associated with dropout from group CBT-I and ultimately lead to strategies for improving retention and optimizing the delivery of group CBT-I in clinic settings.