A Receiver Operator Characteristics Analysis of Treatment Outcome in Binge Eating Disorder to Identify Patterns of Rapid Response

Abstract

The purpose of this study was to perform a receiver operator characteristics (ROC) analysis on a treatment sample from a randomized controlled treatment trial of participants with binge eating disorder (BED). An ROC analysis was completed with 179 adults in a 20-week treatment trial for BED to predict abstinence from binge eating at end of treatment. Percent reductions in binge eating episodes were examined following weeks 1 through 10 of treatment. The rate of percent decrease in binge eating episodes during treatment for BED was a significant predictor of clinical outcome at end of treatment. Participants who demonstrated a 15% reduction in binge eating episodes at week one were more likely to respond positively to treatment and achieve clinical remission. Findings from the current study suggest that a significant reduction in binge eating during the first week of treatment may be predictive of end of treatment remission in those with BED.

Patients diagnosed with binge eating disorder (BED) engage in recurrent episodes of binge eating accompanied by a sense of loss of control. BED is usually a chronic disorder associated with increased rates of obesity, comorbid psychopathology, and decreased quality of life (Pope et al., 2006; Striegel-Moore et al., 2008). A number of treatments have been studied for individuals with BED, including self-help programs, pharmacological interventions, interpersonal psychotherapy (IPT), and cognitive-behavioral therapy (CBT). Psychotherapeutic treatments generally result in clinically meaningful reductions in the number of objective binge eating episodes (OBEs) and generally show better response rates than pharmacotherapies (Wilson, Grilo, & Vitousek, 2007). CBT has been demonstrated to have fairly rapid effects, and currently is the best established treatment for BED (NICE, 2004; Wilson et al., 2007).

Although a number of studies using psychological treatments for BED have been conducted with generally positive results (see review by Vocks et al., 2010), most of these investigations have reported baseline, end of treatment (EOT), and follow-up data. Much less is known about changes in symptom status during the course of treatment. Understanding patterns of improvement, or lack of improvement, during the course of treatment in addition to EOT and follow-up can provide important information about prognosis, attrition, and components of treatment associated with efficacy. Examining trajectories of recovery during treatment may be especially useful in understanding patterns of rapid response: specifically, whether early improvement in symptoms is predictive of treatment outcome.

Several studies have investigated patterns of recovery in other types of eating disorders, particularly bulimia nervosa (BN). For example, a recent study by Marrone and colleagues (2009) used receiver operating characteristics (ROC) analysis to evaluate response to 20 sessions of CBT (delivered via telemedicine or face-to-face) among a group of individuals with BN. Results for the entire sample indicated that the best cut-point for predicting the eventual percent reduction in binge eating occurred during week eight: participants with a 76% or greater reduction in binge eating by week eight were more likely to be abstinent from binge eating and purging at EOT than participants with less than a 76% reduction at week eight. Another study involving 18 sessions of CBT to treat women with BN observed that early response to treatment (reduced purging by 70% by session 6) predicted EOT abstinence (Agras et al., 2000). Similarly, a BN study that included 19 sessions of CBT or IPT with women found that early change in frequency of purging (reduced purging frequency by at least 51% during the first 4 weeks of treatment) was the best predictor of response at EOT and at 8-month follow-up (Fairburn et al., 2004).

Three studies of early rapid response to BED treatment from randomized clinical trials have been published (Grilo & Masheb, 2007; Grilo, Masheb, & Wilson, 2006; Masheb & Grilo, 2007). Grilo and Masheb (2007) examined rapid response (≥ 70% reduction in binge eating by the 4^th week) of 50 patients with BED during a 12-week study. Participants were randomly assigned to one of two treatments: orlistat plus guided self-help CBT or placebo plus guided self-help CBT. The results indicated that 42% of participants experienced a rapid response and were more likely to achieve binge eating remission and 5% weight loss (Grilo & Masheb, 2007).

Grilo and colleagues (2006) used ROC curves to predict outcomes for 108 patients with BED during a 16-week treatment that included four treatment arms: fluoxetine, placebo, CBT plus fluoxetine, and CBT plus placebo. The results showed that 44% of participants who had a rapid response (≥ 65% reduction in binge eating by the 4^th week) were more likely to exhibit greater improvement in eating-disorder psychopathology and greater weight loss than participants without the pattern of rapid response (Grilo, Masheb, & Wilson, 2006). Masheb and Grilo (2007) examined early rapid response (≥ 65% reduction in binge eating by the 4^th week) as a prognostic indicator of treatment outcome in BED. Seventy-five adult patients were randomized to one of two guided self-help treatments: CBT or behavioral weight loss (BWL). The results indicate the rapid responders had a significantly greater reduction in binge eating, overall eating psychopathology and depressive symptomatology than the nonrapid responders. In addition, they found different prognostic significance for the two different treatments: CBT participants did equally well on binge eating (i.e., continued pattern of improvement) regardless of whether or not they had a rapid response, but BWL nonrapid responders were unlikely to improve during treatment (Masheb & Grilo, 2007).

The purpose of the current study was to examine patterns of symptomatic improvement in BED to evaluate whether early reduction in symptoms was associated with binge eating status at EOT using a sample of participants who received group therapy. Understanding the extent to which rapid response is a significant prognostic indicator in group therapy has important clinical implications, particularly given the potential cost effectiveness of group over individual therapy. Identifying which patients are more likely to improve (i.e., fewer binge eating episodes, decreased body-mass-index (BMI), better mood, less depression) based on their pattern of response would be useful in reducing the burden of costs by retaining those helped by group therapy in a less expensive treatment while potentially directing patients who are not likely to be helped to more intensive treatment.

Methods

Study participants

Participants were 18 years and older, of either gender, and were recruited at sites in North Dakota and Minnesota. Institutional review board approval was received from the University of North Dakota, MeritCare Health System, and the University of Minnesota. Written informed consent was obtained from all participants.

The participants in the current study received a manual-based form of CBT (Mitchell et al., 2008) and were randomly assigned to one of three active treatments in which the manual was administered (i.e., therapist-led, therapist-assisted, or self-help), with 15 group sessions (80 minutes each) over a 20-week period (see Peterson, Mitchell, Crow, Crosby, & Wonderlich, 2009). Active treatments were identical except for method of delivery. Participants included 185 adults with a BMI ≥ 25 kg/m² who met full DSM-IV criteria for BED as diagnosed by experienced assessors using the Eating Disorder Examination (EDE; Fairburn & Cooper, 1996). All assessment interviews were audiotaped. Interrater reliability ratings were conducted on 20% of the interviews, yielding intraclass correlation coefficients were .955 to .982.

The frequency of binge eating episodes was measured using the EDE at baseline, end of treatment (or end of waiting period for wait list control), and at 6- and 12-month follow-up assessments. In addition, participants completed binge recall diaries each week to record their binge eating frequency. The participants provided this information during the start of each group session in the clinic.

Height and weight was measured at baseline and end of treatment to calculate BMI. Participants completed the 30-item Inventory of Depressive Symptomatology (IDS) at baseline, midpoint, and end of treatment to assess the severity of depressive symptoms (Rush et al., 1996). Current and past Axis I diagnoses were assessed using the Structured Clinical Interview for DSM-IV-TR (First, Spitzer, Gibbon, & Williams, 2002).

Statistical Analysis

The following research questions were addressed: (1) does early response to treatment predict abstinence at the end of treatment? and (2) does the method of analysis influence the results? The primary aim was to determine the optimal cutoff points for identifying participants who were most likely to respond positively to treatment and to compare differences in response to treatment between responders and nonresponders.

All analyses were conducted using Predictive Analytics SoftWare (PASW) for Windows Version 18.0 (SPSS Inc., Chicago, Illinois). Treatment outcome was defined as abstinence from binge eating episodes for the 30 days prior to treatment. Abstinence was defined as no objective binge eating episodes (OBE) on the EDE in the past 28 days. Early response to treatment was defined as the percent reduction from baseline in binge eating episodes as determined by weekly self-report binge eating diaries. Groups (i.e., rapid responders and nonrapid responders) were compared at baseline on demographic variables using t-test for continuous variables (BMI, age) and chi-square for other variables.

ROC analysis was used to evaluate whether the percent reductions from baseline in binge eating episodes over each of the first 10 weeks of treatment was associated with abstinence at the EOT. The accuracy of the discrimination was assessed using the area under the curve (AUC) value. The AUC represents the probability that a randomly selected abstinent participant will have a greater percent reduction in binge eating episodes than a randomly selected non-abstinent case. The larger the AUC, the higher the accuracy. The best cutoff was determined using the coordinates of the ROC curve so as to jointly maximize sensitivity and specificity. Due to the presence of missing data, analyses were conducted on both the treatment completers as well as the intent-to-treat (ITT) sample. For each of these methods, missing weekly binge eating episodes were analyzed using three different analytic approaches: (1) only those with complete data at a given week were included in the analysis, (2) missing data were imputed using maximum likelihood (ML) estimation, and (3) missing data were imputed using last observation carried forward (LOCF). Data were analyzed for the entire sample and by treatment condition (therapist-led, therapist-assisted, or self-help).

Using the rapid response criteria from Grilo and Masheb (2007; ≥ 70% reduction in binge eating by week 4) and Masheb and Grilo (2007; ≥ 65% reduction in binge eating by week 4), participants in the current study were also separated into rapid responders and nonrapid responders in order to examine patterns of response using these previously established categories. The agreement between this rapid response classification and that based upon the ROC analysis were compared using a kappa coefficient. Changes in weight loss and depression change were compared between rapid response groups using independent samples t-test. The association between rapid response and current and lifetime mood, anxiety, and substance disorders were examined using chi-square tests.

Results

Of the 185 participants randomized to an active treatment group, 179 (97%) participants were included in the analysis. Six participants were not included because they reported zero binge eating episodes at baseline in their weekly binge recall diaries. A total of 121 (65.4%) participants completed the treatment. Mean age of participants was 46.5 years (SD 10.2, range 19 – 65) and 89.9% were women. Most were Caucasian (95.5%) with at least a college degree (31.7%), employed full-time (64.0%), and were taking antidepressant medication (79.3%). Table 1 shows that there were no significant differences between rapid responders and nonrapid responders on pretreatment patient variables (all p-values ≥ .144); with the exception of BMI: rapid responders had higher BMIs at baseline (t = 1.99, df = 170, p = .050).

Table 1.

Pretreatment participant variables by group

Variable	Rapid Responder (n = 81)	Nonrapid Responder (n = 98)	p-value
Mean Age (years)	46.33	47.05	0.645
Mean BMI (kg/m2)	40.49	38.02	0.050
Mean IDS	23.74	24.90	0.506
Current Mood Disorder	8 (9.9%)	10 (10.2%)	>0.999
Lifetime Mood Disorder	53 (65.4%)	54 (55.1%)	0.435
Current Anxiety Disorder	23 (28.4%)	21 (21.4%)	0.495
Lifetime Anxiety Disorder	36 (44.4%)	35 (35.7%)	0.442
Lifetime Alcohol Abuse	24 (26.4%)	23 (28.4%)	0.864
Lifetime Alcohol Dependence	14 (15.4%)	10 (12.3%)	0.661
Lifetime Substance Abuse Disorder	18 (19.8%)	9 (11.1%)	0.144
Lifetime Substance Dependence Disorder	13 (16.0)	16 (16.3%)	0.840

Note: Current alcohol abuse, current alcohol dependence, current substance abuse, and current substance dependence were not included in this table since participants identified with abuse or dependence were ineligible for this study.

Table 2 presents the AUC values for Weeks 1–10 by method for defining treatment outcome (Completer vs ITT) and handling missing weekly binge eating data (complete case vs. ML imputation vs. LOCF). Regardless of the method for defining treatment outcome or handling missing weekly data, the highest AUC values were always at Week 1, indicating that the percent reduction in binge eating episodes from baseline at Week 1 was the best predictor of abstinence at the end of treatment.

Table 2.

Area under the curve (AUC) values and standard errors for the combinations of completed, imputed and last observation carried forward (LOCF) data with completed and intent-to-treat (ITT) samples

	Completer			ITT
Week	Complete	ML	LOCF	Complete	ML	LOCF
1	.669 (.048)	.660 (.048)	.671 (.047)	.676 (.042)	.672 (.042)	.681*(.042)
2	.570 (.050)	.570 (.050)	.570 (.050)	.568 (.045)	.575 (.044)	.575 (.044)
3	.608 (.050)	.608 (.050)	.609 (.050)	.602 (.047)	.604 (.046)	.599 (.046)
4	.591(.051)	.589 (.050)	.579 (.050)	.586 (.047)	.582 (.046)	.582 (.046)
5	.625 (.050)	.599 (.049)	.608 (.049)	.592 (.047)	.581 (.044)	.581 (.044)
6	.647 (.050)	.644 (.048)	.646 (.048)	.631 (.047)	.618 (.042)	.618 (.042)
7	.621 (.051)	.581 (.050)	.611 (.049)	.615 (.048)	.581 (.045)	.619 (.044)
8	.608 (.051)	.586 (.050)	.601 (.050)	.617 (.049)	.611 (.046)	.620 (.046)
9	.579 (.052)	.558 (.051)	.571 (.051)	.576 (.050)	.560 (.047)	.587 (.046)
10	.642 (.050)	.631(.049)	.625 (.049)	.649 (.048)	.652 (.042)	.652 (.042)

ML = maximum likelihood

Table 3 presents AUC values, cutoffs, and sensitivity and specificity for week 1 both overall and by treatment group (therapist-led, therapist-assisted, or self-help). Among the 62 participants assigned to self-help group, 38 (61.3%) completed the treatment; 58 participants were assigned to therapist-assisted group and 40 (69.0%) completed the treatment; finally, 59 were randomized to the therapist-led group and 53 (89.8%) completed the treatment.

Table 3.

Area under the curve (AUC) values, cutoffs, and sensitivity, specificity at Week 1 for the combinations of completed, imputed and last observation carried forward (LOCF) data with completed and intent-to-treat (ITT) abstinence outcomes of at session one among all three treatment groups

		Complete				ITT
	Treatment group	AUC	Cutoff	Sensitivity	Specificity	AUC	Cutoff	Sensitivity	Specificity
Completed	Self-Help	0.699	4.546	0.636	0.63	0.692	4.546	0.636	0.625
	Therapist Assisted	0.739	9.546	0.684	0.7	0.697	15	0.632	0.611
	Therapist led	0.584	18.333	0.567	0.435	0.634	18.333	0.567	0.655
	Overall	0.699	15	0.633	0.629	0.670	15	0.633	0.619
ML	Self-Help	0.699	4.546	0.636	0.63	0.693	4.546	0.636	0.627
	Therapist Assisted	0.706	9.546	0.684	0.667	0.677	21.539	0.579	0.615
	Therapist led	0.584	18.333	0.567	0.565	0.634	15	0.667	0.586
	Overall	0.660	15	0.633	0.629	0.672	16	0.633	0.619
LOCF	Self-Help	0.699	4.546	0.636	0.63	0.692	4.546	0.636	0.353
	Therapist Assisted	0.742	6.331	0.737	0.714	0.704	15	0.632	0.641
	Therapist led	0.584	18.333	0.567	0.435	0.634	15	0.667	0.586
	Overall	0.671	15	0.633	0.634	0.681	12.9167	0.633	0.63

ML = maximum likelihood

At week 1, the highest AUC overall was 0.699, which means that a randomly selected patient who is responding to treatment will have a greater percent reduction in binge eating than a randomly selected patient who is not responding to treatment 69.9% of the time. When we compared the three treatment groups, therapist-assisted group had the higher AUC values than self-help group and therapist-led group, regardless of the combination of the method for defining treatment outcome or handling missing weekly data. The highest AUC value in therapist-assisted treatment group was 0.742, the highest AUC value in self-help treatment group was 0.699, and the highest one in therapist-led treatment group was 0.634.

The best cutoff for percent reduction in binge eating at the end of week 1 for the entire sample was 15%, meaning that participants who showed at least a 15% reduction in binge eating were more likely to be abstinent from binge eating by the EOT than participants who showed less than 15% reduction in binge eating by the week 1. At this optimal cutoff, the specificity and sensitivity have the best balance with sensitivity of 63.3%and specificity of 62.9%. However, when the analysis was completed separately by treatment group, there were differences in the best cutoff for percent reduction in binge eating. For the therapist-assisted group, the optimal cutoff was 6.33%, with sensitivity of 73.7% and specificity of 71.4%. For the self-help group, the best cut point was 4.55%, with sensitivity of 63.6% and specificity of 63.0%; for the therapist-led group, the best cutoff was 18.33%, with sensitivity of 56.7% and specificity of 43.5%.

Defining rapid response as either ≥ 65% or ≥ 70% by the 4^th week of treatment (based upon the Grilo et al. studies) produced identical results in the current sample: n = 141 (73.7%) were classified as rapid responders. When rapid response was defined as ≥ 15% greater reduction in binge eating by the 1^st week of the treatment (based upon ROC analysis), the percentage of rapid responders was 53.1% (n = 81). The kappa coefficient for defining rapid response by these two methods was only 0.196 (p=0.007), indicating low (albeit significant) agreement.

Rapid responders did not differ significantly from nonrapid responders on changes in BMI or depression, regardless of whether rapid response was defined as ≥70% reduction at week 4 or ≥15% or greater reduction at week 1. Based on the 15% cutoff at week 1, rapid responders had significantly greater abstinence rate than nonrapid responders for completer and ITT analysis (p values < .001). However, there were no significant differences between rapid responders and nonrapid responders when the 70% cutoff at week 4 was used. Chi-square tests results showed no significant association between current and lifetime mood, anxiety, and substance disorders and rapid response(all p-values > .300), regardless of the criteria used to define rapid response.

Discussion

The results from the current study extend previous findings that indicate early symptom improvement predicts treatment outcome in patients with BED. These findings suggest that patients who experienced a small, rapid response (defined as 15% reduction in OBEs at the end of the first week) were more likely to have a positive treatment outcome defined as abstinence at the EOT. These findings support several studies by Grilo and colleagues (2006, 2007; Masheb & Grilo, 2007) that classified patients as rapid responders if they reduced binge eating by 65–70% or more during the first 4 weeks of treatment as well as several investigations finding similar results in BN (Agras et al., 2000; Fairburn et al., 2004; Marrone et al.,. 2009).

The current study also compared two classification systems for defining rapid response in BED: the threshold determined by ROC (in this case, 15% reduction in the first week of treatment) and the previously utilized (Grilo et al., 2006; Grilo & Masheb, 2007; Masheb & Grilo, 2007) threshold of 65–70% reduction by week four. Although these contrasting approaches yielded rapid response rates that were correlated, the 65–70% reduction by week four threshold was not associated with remission in this sample. The current study is the only investigation that found the optimal cutoff for defining rapid response was a small reduction early in treatment while all other previous studies of BED and BN observed that the optimal cut-off for defining rapid response was greater than 50% reduction over a longer time period (e.g., four weeks) (e.g., Agras et al., 2000; Fairburn et al., 2004; Grilo et al., 2006; Marrone et al.,. 2009). The inconsistency between the current findings and those of previous studies may be the result of differences in diagnoses (e.g., BED vs. BN), treatment type (e.g., psychotherapy vs. medication), treatment mode (e.g., group vs. individual psychotherapy), treatment delivery (e.g., therapist delivered vs. self-help), or emphasis of treatment (binge eating remission with and without weight loss). Clearly, replication of these studies using various types of patients and treatments is essential to understand patterns of recovery and the extent to which these differ based on diagnostic and treatment types.

Although generally consistent with previous findings of rapid response in BED, several discrepancies between the results of this study and those of previous investigations are notable. First, in contrast to previous studies by Grilo and colleagues (2006, 2007), rapid response in the current study was associated with binge eating remission but not significant weight loss. This difference may be due to the fact that interventions in the previous studies focused on binge eating and weight loss and the intervention in the current study did not explicitly emphasize weight loss. In addition, Masheb & Grilo (2007) found that rapid response was associated with a reduction in psychopathology as well as binge eating; in the current study, rapid response was not correlated with improvements in co-occurring depressive symptoms. Whether the inconsistencies between the current and previous studies in the scope of rapid response in BED are the result of differences among the treatments is unclear and worthy of future study.

This study has several strengths, including the fact that the data were based on a randomized controlled trial, participants were diagnosed and measured at outcome using the EDE, and the analyses were examined using various methods and assumptions (e.g., treatment completers vs. ITT) to determine the robustness of findings. However, several limitations are noteworthy and should be considered in interpreting these findings. First, the examination of different treatment conditions as well as attrition across all treatment groups led to limitations in sample size that may have compromised power. Second, frequency of binge eating during treatment was based on self-reported questionnaire data. Because these data are based on self-report rather than clinical interview, the extent to which these episodes of binge eating would have met the DSM-IV criteria for being clearly large in size and accompanied by loss of control is uncertain. Although conducting weekly or daily EDE interviews to assess these aspects of binge eating would have been impractical in a treatment outcome study such as this one, participants’ self-reported binge eating frequency may not correspond with the EDE which was used as the primary outcome measure in this investigation. Finally, although attrition made such analyses impossible in this sample, these findings are based on short-term treatment outcome only; whether the 15% reduction in week one is a significant predictor of longer term outcome is not known.

Patients in the current study who reported an early response by the end of the first week tended to have better outcomes than those who did not respond as quickly to treatment. This finding has important clinical implications because knowing the likelihood of outcome based on the initial treatment response could provide valuable information for making treatment decisions. For example, patients who fail to report an early response may consider alternative treatments, such as adding medication, to augment CBT. Alternatively, those patients who show initial response to group therapy can be retained in a treatment that is potentially less costly than individual psychotherapy. Understanding early patterns of recovery can ultimately facilitate matching patients to treatments that are most likely to be efficacious for them.

Abbreviations

BED

binge eating disorder

ROC

receiver operator characteristics

IPT

interpersonal psychotherapy

CBT

cognitive-behavioral therapy

OBEs

objective binge eating episodes

EOT

end of treatment

BN

bulimia nervosa

EDE

Eating Disorder Examination