Interrater reliability of the Eating Disorder Examination among post-bariatric patients

Abstract

Background:

Clinical assessment of eating behaviors with patients who undergo bariatric surgery is challenging due to the complexity of symptom presentation post-operatively. The Eating Disorder Examination (EDE) is a widely-used semi-structured clinical interview of eating-disorder psychopathology, yet no studies have examined the interrater reliability among postoperative bariatric surgery patients.

Objectives:

The present study aimed to examine the interrater reliability of the EDE, and an alternative classification of size-specific thresholds of binge-eating episodes in a postoperative bariatric surgery sample.

Setting:

University School of Medicine, United States

Methods:

Participants interviewed were a randomly selected subset (N=20) from a consecutive series of adults seeking treatment for eating concerns following bariatric surgery. Audio-taped interviews were rated independently by one of four expert raters. Interrater reliability was assessed using intra-class correlation coefficients (ICC) and kappa statistic.

Results:

ICCs for the original four EDE subscales were excellent, ranging from 0.88 to 0.98. ICCs for the alternative brief three subscales were also excellent, with a range of 0.78 to 0.97. ICCs for bariatric-LOC eating episodes were in the good to excellent range, with a range of 0.66 to 0.99. Kappa agreement for bariatric overeating episodes was moderate (0.60).

Conclusions:

These findings, based on four expert raters, suggest that complex eating-disorder psychopathology, as well as the newly-proposed eating behavior with size thresholds relevant to bariatric patients, can be reliably assessed. This is the first study to provide initial evaluation and support for the interrater reliability of the original EDE with additional modified eating categories developed for post-bariatric surgery patients.

Introduction

While bariatric surgery offers the most effective and durable long-term weight loss for the treatment of obesity and co-morbid medical sequalae, outcomes remain variable [1]. Certain eating behaviors (i.e., loss-of-control eating) and eating-disorder psychopathology (e.g., maladaptive forms of restraint, shape, weight, and eating concerns) are significant predictors of poorer post-operative outcomes [2-4]. Psychometrically-valid and reliable assessment measures in the pre- and post-operative bariatric population are required to assess disordered eating and related psychopathology adequately and to examine the relationship between these symptoms and surgical outcomes.

In general, available instruments for the assessment of eating behaviors in bariatric surgery patients have significant limitations in terms of their psychometric properties [5-7]. Instruments such as the Eating Disorder Examination (EDE) [8], a widely-used semi-structured clinical interview designed to assess eating-disorder psychopathology, were originally designed specifically for the assessment of eating disorders (anorexia nervosa and bulimia nervosa) among non-surgical populations. Psychometric studies of the EDE-Questionnaire, a self-report version of the EDE interview [9-11], with data obtained from bariatric populations have yielded mixed findings with regard to both factor structure and reliability. Moreover, research comparing EDE interview versus self-report methods have found low levels of agreement in studies performed during pre- [12] and post-operative periods [7]. Thus, these measures in their original form may not accurately nor reliably assess eating-disorder psychopathology as experienced by bariatric surgery patients [13].

The clinical assessment of eating behaviors in bariatric surgery patients can present difficulties due to the complexity and variability of symptom presentation post-operatively. One major challenge is the assessment of binge-eating episodes. As defined by the Diagnostic and Statistical Manual of Mental Disorders – Fifth Edition (DSM-5; [14]), and also as adopted in the EDE, the two essential required criteria for binge eating are (1) experiencing a sense of loss-of-control (LOC) while (2) eating an unusually large amount of food. However, ambiguities about definitions for an “unusually large amount,” which can make this assessment difficult in any patient group, are even more challenging when applied to patients post-bariatric surgery [15] where the gastric limitations imposed by the surgery prevent most patients from consuming unusually large quantities of food, especially within the first year after surgery. Not surprisingly, one study found that patients denied any objective (i.e., “unusually large amount”) binge-eating episodes (OBEs) six months after surgery [16], and full-threshold diagnoses of BED (requiring OBEs) did not emerge until two years following bariatric surgery and seemed quite uncommon [17]. If the focus of the EDE assessment remains on OBEs and full-threshold BED during the post-operative period, the majority of cases with disordered eating involving LOC-eating will go undetected.

Refinement of binge-eating categories might be necessary to determine binge-eating thresholds for bariatric patients (i.e., what constitutes an “unusually large” quantity of food for a post-operative bariatric patient). Indeed, a recent study developed a new conceptualization of binge-eating episodes, based on size-specific thresholds for patients who underwent bariatric surgery [18]. This study supported the utility of size-specific thresholds for binge-eating, as loss-of-control eating size was related to broad levels of psychopathology and disability in a post-surgical sample. The reliability of these proposed size-specific thresholds for binge-eating/LOC-eating among patients who undergo bariatric surgery need to be examined alongside reliability for the associated behavioral/cognitive features of eating-disorder psychopathology.

Thus, the present study aimed: 1) to fill the important gap in the literature by examining interrater reliability of the EDE for the original widely-used factor structure and for an alternative factor structure recently reported and replicated with superior psychometrics [10, 11] among post-operative bariatric surgery patients and 2) to examine interrater reliability of newly developed post-bariatric size-specific thresholds for binge-eating and overeating [18] for use in the post-operative period (See Figure 1 for the original EDE eating behaviors and additional modifications of eating behaviors for bariatric patients).

Figure 1.

Schematic representation of overeating and binge-eating constructs adapted and expanded for assessment of patients following bariatric surgery.

Note. ^aB-OBE: defined as consuming greater than 1 cup 5-8 months post-operatively or 2 cups greater than 9 months post-operatively. ^bB-SBE: defined as consuming less than a cup 5-8 months post-operatively, or less than 2 cups nine months post-operatively. ^cB-OOE: defined as consuming greater than 1 cup 5-8 months post-operatively, or 2 cups nine months post-operatively without experiencing LOC.

Method

Participants and Procedures

Participants interviewed were a randomly selected subset from a consecutive series of individuals seeking treatment for eating concerns following bariatric surgery. Participants were assessed approximately 6 months after bariatric surgery, as part of a larger treatment study; this time-frame follows prior research findings that LOC eating emerges, on average, approximately 6 months after surgery and predicts poorer weight loss outcomes [2]. Participants were eligible to take part in the study if they were between 18 and 65 years old and reported LOC eating (e.g., a subjective sense of LOC while eating), regardless of size. Exclusion criteria included use of any current medications known to influence weight or eating, substance use dependence, or severe psychiatric illness requiring immediate treatment. The interviewees (N=20) were predominately female (n=14; 70%) and the majority were White, Not Hispanic (n=13; 65%) and on average had a current BMI of 39.88 (SD = 9.16). The mean age of the subsample was 45.15 (SD = 9.63) years. Interviews took place approximately six months (M = 6.33, SD = 1.52; range 4 - 9 months) after gastric bypass or sleeve gastrectomy surgery performed at the Yale Bariatric/Gastrointestinal Surgery Center of Excellence.

Twenty audio-taped interviews, randomly selected from a larger consecutive series of baseline treatment assessments, were rated independently by one of four expert raters, one of whom was the developer of the original EDE. This study received approval from the Yale University School of Medicine Institutional Review Board and all participants provided written informed consent.

Measures

Participants were assessed with the Eating Disorder Examination-Bariatric Surgery Version (EDE-BSV), a semi-structured interview assessing eating-disorder symptomology and overeating behaviors, adapted for bariatric surgery patients [19-21]. To date, there have been no studies examining the psychometric properties or factor structure of the EDE-BSV interview. The EDE-BSV contains all the standard EDE questions yielding a global score, as well as the four EDE subscales (i.e., restraint, eating concern, weight concern, shape concern). Scores on the EDE subscales range from 0-6, with higher scores reflecting greater eating-disorder psychopathology. The EDE is hypothesized to have a 4-scale structure with the following subscales: restraint over eating, eating concern, shape concern, and weight concern yet an alternative briefer 3-scale structure (overvaluation of shape/weight, dissatisfaction with shape/weight and restraint) has superior psychometric properties in both surgical and non-surgical samples [11, 22]. The original EDE subscales and global scores were evaluated. In addition, given the support for the alternative brief three-scale factor structure of the EDE among bariatric surgery candidates [10, 11], a finding that has also received consistent support in studies with patients with restrictive eating disorders (anorexia nervosa and bulimia nervosa) [23] and for patients with BED [24] and across non-clinical samples [22], the alternative brief structure comprising three subscales (restraint, overvaluation of shape and weight, and dissatisfaction of shape and weight) was also examined. The EDE also assesses overeating behaviors, including objective binge-eating episodes (OBE; eating an unusually large amount of food while experiencing a subjective sense of LOC while eating), subjective binge-eating episodes (SBE; eating an amount not considered unusually large while experiencing a sense of LOC, regardless of perceived size), and Objective-Overeating Episodes (OOE; eating an unusually large amount of food without experiencing LOC). In the present study, an SBE was considered present if LOC was endorsed while eating small or typical amounts, regardless of perceived size. The EDE assessment of objective overeating is consistent with DSM-5 criteria of BED which defines binge-episodes as an unusually large amount of food consumed within a discrete period of time (i.e., within two hours).

Additional items designed to assess bariatric-LOC eating episodes with respect to size-specific thresholds in terms of the amount of food consumed were created in a prior study [18]. These size-specific thresholds were based on recommended guidelines of portion sizes for a patient who has undergone bariatric surgery based on the Yale Bariatric/Gastrointestinal Surgery Center of Excellence and from guidelines within the EDE which generally regard traditional binge-eating episodes as unequivocally large in terms of the overall amount, large for a particular type of food, or an excessive amount of food given the context. For a patient who is 5-8 months post-surgery, the amount of food recommended during an eating occasion is half a cup or less of food, while up to one cup of food is recommended to patients who are nine months or more post-surgery. Thus, three new separate categories of “bariatric binge-eating episodes” based on the amount of food consumed during an eating episode were evaluated. Episodes of LOC eating were considered objectively large for a post-bariatric surgery patient if the quantity of food was greater than a cup (for patients 5-8 months post-surgery), or greater than two cups (for patients greater than or equal to nine months post-surgery). These were classified as “Bariatric-Objective Binge Episodes” (B-OBE). Episodes of LOC were considered subjectively large for a post-bariatric surgery patient if the quantity of food was less than a cup (for patients 5-8 months post-surgery) or less than two cups (for patients greater than or equal to nine months post-surgery). These were classified “Bariatric-Subjective Binge Episodes” (B-SBE). Additionally, eating episodes during which more than a cup of food was consumed (for patients who were 5-8 months post-surgery), or more than two cups of food were consumed (for patients greater than or equal to nine months post-surgery), without experiencing LOC, were defined as “Bariatric-Objective Overeating Episodes” (B-OOE). For example, consuming three cups of popcorn within a two-hour period while experiencing a sense of LOC would be categorized as a B-OBE, whereas consuming two small pieces of candy while experiencing a sense of LOC would be categorized as a B-SBE. See Figure 1 for a matrix of eating episodes assessed in the present investigation.

Statistical Analyses

Interrater reliability was assessed using intra-class correlation coefficients (ICC) for the original four-subscales and the alternative three-subscales, as well as bariatric-LOC eating episodes including B-OBEs and B-SBEs. Kappa statistic was calculated for Bariatric Overeating Episodes (i.e., B-OOEs) as endorsement of B-OOEs was infrequent (M = 1.25, SD = 2.49, range = 0 to 8 episodes, with 75% endorsing no B-OOEs). Given that participants in this study were assessed approximately six months post-operatively, we anticipated a low frequency of objectively unusually large episodes (OBEs and OOEs) as defined by the original EDE items and thus did not include them in the current study of reliability.

Results

Participants, on average, endorsed having 30.65 (SD = 47.90) episodes of LOC eating (i.e., OBEs, and/or Bariatric-LOC eating episodes combined) during the past 28 days. Endorsement of OBEs (M = 1.20, SD = 3.32) and OOEs (M = 0.10, SD = 0.45) during the past 28 days were infrequent: 80% of participants reported no OBEs, while 95% endorsed having no OOEs during the previous 28 days. Among the 20% (n=4) who reported OBEs, the frequency ranged from one to thirteen episodes during the past 28 days. Only one participant reported OOEs endorsing two episodes. Thus, ICCs were not calculated for OBEs or OOEs as they were so infrequent.

Table 1 describes the means and standard deviations for the original rater, the interrater reliability, and confidence intervals for EDE subscales, the alternative three-scale structure of the EDE, and newly developed post-bariatric size-thresholds for bariatric-LOC eating episodes and bariatric overeating episodes.

Table 1.

Means, Standard Deviations, and Interrater reliability of EDE Scales and Overeating Behaviors

	Mean	SD	ICC	ICCa	CI
Original EDE Scales
Dietary restraint	1.92	1.25	0.98**	Excellent	0.94-0.99
Eating concern	1.36	0.84	0.93**	Excellent	0.84-0.97
Weight concern	2.62	1.23	0.88**	Excellent	0.73-0.95
Shape concern	2.43	1.59	0.91**	Excellent	0.78-0.96
Global score	2.08	1.09	0.95**	Excellent	0.89-0.98
Alternative EDE Scales
Restraint	2.97	1.86	0.97**	Excellent	0.92-0.99
Overvaluation	2.45	1.86	0.82**	Excellent	0.60-0.92
Dissatisfaction	3.08	1.58	0.78**	Excellent	0.52-0.91
Global score	2.83	1.40	0.83**	Excellent	0.83-0.97
Bariatric-LOC Eating Episodes
Bariatric-Objective binge episodes	10.70	10.85	0.66**	Good	0.28-0.86
Bariatric-Subjective binge episodes	18.75	43.34	0.99**	Excellent	0.99-0.999b
	Mean	SD	Kappa	Kappaa	% Endorsed
Bariatric Overeating Episodes
Bariatric-Objective overeating episodes	1.25	2.49	0.60	Moderate	25

Note. EDE = Eating Disorder Examination; ICC = Intraclass correlation. N=20 for first and second rater; except for B-OOE, n=17 of second rater

^*p<0.01

^**p≤0.001

^aCiccheti and Sparrow (ICC) [25]; Viera and Garrett’s guidelines (Kappa) [26].

^bcValues were not rounded above .999.

ICCs within 0.75 to 1.00 were regarded as excellent, .60 and .74 as good, 0.40 to 0.59 as poor, and below 0.40 as unacceptable [25]. ICCs for the original four subscales were excellent, ranging from 0.88 to 0.98 with 0.95 for the overall EDE Global Score. ICCs for the alternative brief three subscales were also excellent, with a range of 0.78 to 0.97, and 0.83 for the alternative factor brief global score. The ICC for the B-OBEs was good (0.66) and B-SBEs was excellent (0.99). To interpret kappa statistic, we used Viera and Garrett’s guidelines [26]. Kappa agreement for B-OOEs was moderate (0.60).

Discussion

This is the first study to provide initial evaluation and support for the interrater reliability of the original EDE with additional modified eating categories developed for post-bariatric surgery patients. Overall, the reliability coefficients were consistently in the excellent range across all four original EDE subscales and global scale, as well as across the alternative EDE three-scale structure and its corresponding EDE global scale. Moreover, reliability coefficients of the newly-proposed bariatric-LOC eating episodes assessed in the present study, ranged from good to excellent. Our findings, based on four expert raters, suggest that the complex eating behavior and eating-disorder psychopathology of these patients can be assessed reliably. Replication by less expert raters requires investigation.

Findings for interrater reliability were similar to those of previous studies examining the interrater reliability of the original EDE across a variety of clinical and non-clinical samples [6] including research specifically with BED [27]. In this study, the interrater reliability based on the original four-subscale structure was similar to the alternative three-subscale scale structure; reliability coefficients were all within the excellent range. Researchers wishing to reduce participant burden might opt to use the abbreviated version of the EDE with the three-scale structure, which is composed of only 7 items for eating-disorder psychopathology in addition to the eating behavior items.

The second aim of the present investigation was to evaluate the interrater reliability of post-operative size thresholds relevant for post-bariatric surgery patients. The interrater reliability of the bariatric-LOC eating episodes evaluated in the present investigation varied by type, although coefficients ranged from good to excellent. Notably, the reliability coefficient of the bariatric-subjective binge episodes (B-SBEs) was strong. Reliability of bariatric-objective binge episodes (i.e., more than one cup, or two cups of food, 5-8 months, or nine months postoperatively, respectively) was good. Kappa statistic of the bariatric overeating episodes (i.e., without LOC endorsed) was moderate. These findings suggest that the bariatric post-operative eating behaviors proposed in the present study were assessed reliably; however, ratings of bariatric-objective overeating behaviors (B-OOEs) were relatively less consistent than the bariatric binge-eating behaviors across raters, though perhaps impacted by the overall lower frequency of B-OOEs endorsed from this participant sample. Taken together, findings from the present study provide initial support of one estimate of reliability of the EDE among post-bariatric surgery patients, in addition to the recent findings supporting for the clinical utility of assessing size-specific thresholds of eating episodes [18]. These findings are consistent with the recent changes in the ICD-11 regarding assessing various forms of binge eating [28].

Several strengths and limitations of the present investigation are noted with implications for future research. Primary strengths include the use of expert raters and the examination of both the original and the brief alternative factor structures of the EDE. Another strength is the development and testing of bariatric-specific size thresholds for objective and subjective binge-eating episodes among individuals with LOC eating after bariatric surgery. All participants endorsed at least weekly LOC eating, which allowed for examination of the theoretically-proposed bariatric eating episodes unique to post-surgical patients. Additionally, the episodes of bariatric binge eating and overeating were specifically tailored to examine size-thresholds depending on individuals’ time course in the post-operative period. Nonetheless, generalizability of our findings may be limited to treatment-seeking groups and those interested in research participation. Participants in this study were also assessed relatively soon after bariatric surgery and future studies should examine interrater reliability later in the post-operative period (e.g., 12-24 months) as well. It is important to note that the proposed and tested size thresholds for bariatric binge eating and overeating episodes in the present study are simply one approach to examining potentially problematic post-surgical eating episodes that were evaluated in a prior study [18]. These thresholds were determined based on nutritional guidelines established at one bariatric surgery center of excellence and general guidelines of the EDE. There might be other ways to define the size of a “bariatric binge” not considered by the investigation; research is needed to determine consensus for size thresholds for such eating after bariatric surgery (e.g., laboratory eating studies).

Notably, the present study was unable to measure interrater reliability of objective bing-eeating episodes with confidence given that, as expected, so few individuals reported objectively large binge-eating episodes six months after surgery. Reliable assessment of the quantity of food consumed during a binge might be clinically useful for reasons not examined in the present study. For instance, eating an unusually large amount of food after bariatric surgery might have unique consequences, such as plugging or dumping syndrome; however, further research is needed to better understand the physical consequences of binge-eating following bariatric surgery. Similarly, further examination of interrater reliability of objectively large binge-eating episodes in a larger and more diverse sample of post-operative patients including both sleeve gastrectomy and Roux-en-Y gastric bypass surgeries is warranted. Finally, interrater reliability was the only form of reliability examined in this investigation. Examining other psychometric properties of the EDE within post-bariatric surgery patients is warranted.

In conclusion, the present findings lend support for one aspect of reliability of the EDE with post-bariatric surgery patients. Our findings suggest that the original and alternative EDE scales, as well as the newly proposed size-thresholds for eating behaviors relevant to bariatric patients, can be evaluated reliably among experts. Future research should examine the clinical prognostic significance of these newly proposed post-operative eating categories and further evaluate the psychometric properties

Highlights.

• Reliable assessments of complex eating after bariatric surgery are warranted

• The Eating Disorder Examination (EDE) assesses eating-disorder psychopathology

• Psychometric studies of the EDE among postoperative bariatric patients are limited

• Support for the interrater reliability of the EDE in bariatric patients was found