Rapid Assessment of Reward-Related Eating: The RED-X5

Abstract

Objective:

Obesity’s prevalence has created a plethora of questionnaires characterizing psychological aspects of eating behaviour, such as reward-related eating (RRE). The Reward- based Eating Drive questionnaires (RED-9, RED-13) broadly and deeply assesses the RRE construct. However, large-sample research designs require shorter questionnaires that capture RRE quickly and precisely. We sought to develop a brief, reliable, and valid version of the RED questionnaire.

Methods:

We used all-subset correlation to find a subset that maximally associated with the full RED-13 in two separate samples. We validated results in a third independent sample. We also assessed internal consistency, test-retest reliability, and ability to explain variance in external outcomes.

Results:

A 5-item questionnaire (RED-X5) correlated strongly with RED-13 in the independent sample (r = .95). RED-X5 demonstrated high internal consistency (omega total > .80) and 6- month test-retest reliability (r = .72). RED-X5 accurately reproduced known associations between RED-13 and body mass index, diabetes status, and craving for sweet and savory foods. As a novel finding, RED questionnaires predicted laboratory intake of chips.

Conclusions:

The RED-X5 is a short, reliable, and valid measure of the RRE construct and can be readily implemented in large-sample research designs where questionnaire space is limited. Materials are available at https://osf.io/bd3mg/.

Introduction

Obesity accounts for more than 28% of annual US healthcare spending, and affects nearly 40% of US adults (1, 2). The rising prevalence of obesity is at least partially attributable to the modern food environment, which is replete with foods engineered to be highly palatable (rewarding) and thus overconsumed (3–5). Indeed, studies of US adult populations highlight the common experience of difficulty controlling their eating of highly rewarding foods (6–8).

This difficulty can be captured by a plethora of self-report measures that assess reward- related eating (RRE), defined as eating driven by the rewarding and relieving aspects of highly palatable food (9) and uncontrolled eating (UE), the tendency to eat more than necessary because of a loss of control over intake (10). Both RRE (11, 12) and UE (13) correlate with body mass index (BMI). The constructs of RRE and UE share significant overlap, as individuals are more prone to experience loss of control over eating in the presence of highly rewarding food, which is frequent in the modern obesogenic environment. Most questionnaires that assess UE and RRE tend to be lengthy, ranging from 9 items (Reward-based Eating Drive Scale; RED-9; (11) to 51 items (Three Factor Eating Questionnaire; TFEQ; (14). RRE and UE appear useful in explaining psychological processes that correlate with and predict BMI and food intake (15, 16), and RRE was recently highlighted as a core psychosocial measure for obesity treatment trials (ADOPT initative, 15). We therefore sought to develop a brief version of the RED scale (RED-X), that would maximally capture variability in these overlapping constructs and can be used in large panel studies where space and time are limited.

RED scale is one of the many self-report measures of eating behavior that tap into RRE and UE. RED exists in 9- and 13-item formats (11, 12). The RED scales assess three related constructs: lack of control over eating, lack of satiety, and preoccupation with food, and include items based on existing questionnaires, namely the Binge Eating Scale (17) and the TFEQ (14), as well as newly developed items. RED-9 scores correlate with BMI cross-sectionally and predict change in BMI over time (11). Recent data have shown that weight-loss interventions may lead to weight-loss via reductions in RRE as indexed by RED-9 (18). Additionally, higher RED-9 scores have been associated with greater daily craving intensity; however, on days when participants received a medication that dampened craving (naltrexone), this association was reduced (19).

When shortening the RED scale, the short version cannot capture the full construct as well as the RED-13 (12). This is called the bandwidth-fidelity dilemma (20, 21). In that dilemma, the desire to maximize construct coverage (bandwidth) is hampered by practical limits of questionnaire length. Such limits force the questionnaire developers to focus on measurement precision (fidelity) on a certain part of the continuum. For RED-X, a typical use case will be correlating the short form with other anthropometric, psychological, or genetic variables. As most people are likely to fall in the middle ranges of the scale, the RED-X will maximally discriminate among people in the middle spectrum of RRE. We therefore opted for an all-subset correlation method (reviewed in 22) as the brief scale should maximally recreate the normal distribution established with its longer version.

This report summarizes the data-driven process we employed to develop a shortened version of the RED scale (RED-X). We used multiple techniques to ensure that the RED-X maintained good measurement properties, related well to multiple external criteria, and maximally discriminated among people in the normal distribution. To do so while maximizing generalizability across contexts, we used datasets collected both online (Amazon Mechanical Turk [MTURK]) as well as in-person (US community sample of employees at a major university, and Canadian undergraduates at a major university). Additionally, to maximize external validity, we used variables associated with long-term health outcomes, such as in-laboratory eating behavior, diabetes status, BMI, and food cravings (12, 23, 24).

Methods

Participants

Participant data were drawn from four datasets: (1) an MTurk sample (Web1, n = 349), (2) a second MTurk sample (Web2, n = 346), (3) a US community sample (US community; n = 106) of adults who completed an in-person survey, and (4) a Canadian university sample (n = 165 total, n = 51 for the in-laboratory sample who completed the laboratory-based eating procedure).

Procedures

In studies for samples Web1, Web2, and US community, the University of California, San Francisco Institutional Review Board (IRB) approved of all procedures, and all participants provided written informed consent. Surveys were administered through the Research Electronic Data Capture (RedCap) survey system (25). Further details on samples Web1 and Web2 can be found in Mason et al (12). The US community sample (n = 106) was collected in the context of an ongoing change in UCSF foodservice policies. Participants completed the items on as part of a baseline survey battery and then again 6 months later.

The Canadian sample was collected as part of a larger brain imaging and transcranial magnetic stimulation study on self-control. Participants responding to an online advertisement underwent a phone screening with the following criteria: they needed to live in Canada for most of their lives (to ensure sufficient and a similar amount of exposure to the food stimuli presented to them during the experiment amongst participants); they should be trying to lose or maintain their weight and/or practicing healthy eating; they should not be on any diet that restricts certain foods (e.g., vegetarians); no pregnancy; no use of recreational drugs; no medications that may affect brain functioning (contraceptives were allowed) ; no physical, neurological or psychiatric illness; no personal or family history of epilepsy; no metals in the body including permanent braces; no claustrophobia. 165 participants passing the screening procedure filled out online questionnaires, which included RED-18 (a preliminary version of RED-13 (12)), and selfreported height and weight. Based on the score on the RED-18, only those classified as “high self-controllers” (below 0.5SD in RED-18) and “low self-controllers” (above 0.5 SD in RED-18) were selected to participate in the follow-up experiment. The extreme-groups design was used to maximize the potential to observe group differences in the brain imaging study. The experiment was approved by the Montreal Neurological Institute Research Ethics Board and entailed functional magnetic resonance imaging, transcranial magnetic stimulation, electroencephalography, as well as an in-laboratory food intake. Note that the transcranial magnetic stimulation sessions were performed after the survey data were collected. The brain imaging procedures will be described in a future publication.

Measures

Participants completed self-report measures (all samples) and a laboratory-based eating task (Canadian university sample only).

Candidate item pool.

We considered candidate items from the RED-13, an extension of the RED-9 (12). The RED-13 includes 4 additional items identified as candidates to expand the scale (which all maintained at least a 0.45 correlation with the RED-9 general factor). See Table 2 for the items. The RED-13 comprises items from the following sources.

Table 2.

Item pool (RED-13)

Item	Origin	Subscale
I feel out of control in the presence of delicious food	RED1	LOC
* When I starteating, I just can’t seem to stop	RED2	LOC
If it difficult for me to leave food on my plate	RED3	LOC
* When it comes to foods I love, I have no willpower	RED4	LOC
I get so hungry that my stomach often feels like a bottomless pit	RED5	LOS
* I don’t get full easily	RED6	LOS
It seems like most of my waking hours are preoccupied by thoughts about eating or not eating	RED7	PWF
* I have days when I can’t seem to think about anything else but food	RED8	PWF
*Food is always on my mind	RED9	PWF
I feel hungry all the time	TFEQ39	LOS
I find myself continuing to consume certain foods even though I am no longer hungry	YFAS2	LOC
I can’t stop thinking about eating no matter how hard I try	FCQTR10	PWF
If food tastes good to me, I eat more than usual	DEBQ2	LOC

^*Note. Item in final RED-X5. RED = Reward-based Eating Drive scale; TFEQ = Three Factor Eating Questionnaire; YFAS = Yale Food Addiction Scale; FCQT = Food Craving Questionnaire, Trait, Reduced; DEBQ = Dutch Eating Behavior Questionnaire; LOC = Loss of control; PWF = Preoccupation with food; LOS = Lack of satiety. The RED-X5 questionnaire can be freely used when attributed to.

Reward-based Eating Drive Scale (RED-9 (11)).

The RED-9 assesses three dimensions of reward-related eating: loss of control over eating, lack of satiety, and preoccupation with food. Of the 9 items, 2 items originate in the Binge Eating Scale (BES; (17)), 4 items originate in the Three Factor Eating Questionnaire (TFEQ; (14)), and 3 items were developed for this scale. Sample items include, “When I start eating, I just can’t seem to stop” (lack of control), “I don’t get full easily “ (lack of satiety), and “Food is always on my mind” (preoccupation with food). Participants answered on a 5-point scale (1 = strongly disagree to 5 = strongly agree). Total scores for this sample were computed by averaging all items. Higher scores reflect higher reward-based eating drive.

Three Factor Eating Questionnaire (TFEQ; (14)).

The 51-item TFEQ comprises three subscales. The 20-item cognitive restraint subscale (TFEQ-R) assesses conscious mechanisms for restraining food intake. The 20-item disinhibition subscale (TFEQ-D) assesses abilities to control one’s eating. The 15-item hunger subscale (TFEQ-H) assesses hunger and its behavioral consequences.

Yale Food Addiction Scale (YFAS; (26)).

The 25-item YFAS assesses food addiction symptoms based on the 7 symptoms of substance dependence articulated in the DSM-IV-TR (e.g., withdrawal, tolerance, continued use despite problems; (27)). Participants respond on scoring schemes that include dichotomous and frequency scoring (e.g., ranging from never to four or more times daily).

Food Craving Questionnaire, Trait, Reduced (FCQTR; (28)).

The 15-item FCQ-T-R assesses (1) preoccupation with food, i.e., obsessive thoughts about food and eating, (2) loss of control over eating, i.e., difficulty regulating eating behavior when exposed to food cues, (3) positive outcome expectancy, i.e., believing that eating is positively reinforcing, and (2) emotional craving, i.e., the tendency to crave food when experiencing high levels of emotion. Items are answered on a 6-point scale from 1 (never) to 6 (always). In this study, all items were responded to on a scale from 0 (strongly disagree) to 5 (strongly agree).

Dutch Eating Behavior Questionnaire (DEBQ; (29)).

This 33-item scale comprises three subscales. The 10-item Restraint subscale (DEBQ-R) assesses dietary restraint, also termed cognitive restraint. The 10-item External Eating subscale (DEBQ-X) assesses the tendency to eat in response to external food-related cues such as the sight, taste, and smell of attractive food. The 13-item Emotional Eating subscale (DEBQ-E) assesses eating triggered by specific and diffuse emotions such as anger, boredom, anxiety, or fear.

Outcomes.

We collected self-report data on craving for particular foods (sweet versus savory) as well as health status (diabetes) and anthropometrics, and in one sample, assessed laboratory-based eating behavior.

Craving for sweet and savory foods.

Participants in the Web2 sample self-reported cravings for sweet and savory foods on the Control of Eating Questionnaire (CoEQ; (30)). Participants completed two subscales that specifically assess food craving: One 4-item subscale assesses cravings for sweet foods, and another 4-item subscale assesses cravings for savory foods. Representative items are, “How often have you had cravings for sweet foods (cakes, pastries, biscuits, etc.)?” and “How often have you had cravings for starchy foods (bread, pasta)?” Items are answered on a visual analog scale with anchors that go from 1 (not at all strong/not at all) to 100 (extremely strong/extremely often). We computed total scores for each subscale as the mean of items for that scale, with higher scores indicating stronger/greater craving.

Laboratory-based eating behavior.

In the Canadian study, the food intake procedure was conducted on Day 1 of the experiment following brain imaging, and at least 4 hours of fasting. Participants were offered a bowl of Lays potato chips of their favorite flavor for a period of 30 minutes while they filled out other questionnaires not analyzed here. They were not told that their chip consumption would be recorded. Potato chips consumption was measured in grams by weighing the bowl before and after the session.

Type 2 diabetes (T2DM).

Participants in the Web 2 sample self-reported whether they had been diagnosed with T2DM by a medical professional.

Demographics and anthropometries.

Participants reported their age (years), biological sex, educational attainment, race/ethnicity, and total annual household income. We computed BMI from self-reported height and weight. See Table 1.

Table 1.

Participant characteristics for all studies.

Sample
Variable	MTurk 1	MTurk 2	US community (baseline / followup)	Canadian University Students (online / in lab)
N	349	346	106 / 94	165/51
Age: M (SD)	34.23	35.43	42.34 (11.82)	-
	(10.6)	(11.04)
Body Mass Index: M	26.4 (6.72)	25.65 (6.6)	29.06 (6.66) /	22.6
(SD)			29.08 (6.44)	(3.15)
Race: N (%)				-
White	231	243	28 (26.42%)
	(66.2%)	(70.2%)
Black	15 (43.0%)	26 (7.5%)	13 (12.26%)
Asian / Pacific	64 (18.3%)	41 (11.9%)	32 (30.19%)
Islander
Hispanic / Latino	25 (7.2%)	25 (7.2%)	15 (14.15%)
Native American /	2 (0.6%)	2 (0.6%)
Alaska Native
Mixed Race	2 (0.6%)	9 (2.6%)	18 (16.98%)
Declined Response	0 (0.0%)	0 (0.0%)	0 (0.0%)
Education: N (%)				-
Some High School	35 (10%)	1 (0.3%)
High School	77 (22.1%)	42 (12.1%)
Diploma
Some College	-	100 (28.9%)
Associates Degree	45 (12.9%)	33 (9.5%)
(AS, AA)
Bachelor’s Degree	150	137
(BA, BS)	(43.0%)	(39.6%)
Advanced Degree	40 (11.5%)	32 (9.25%)
(MA, MS, MD, PhD,
JD)
No Schooling	2 (0.6%)	1 (0.3%)
Female: n (%)	137	167	60 (56.6%)	106
	(39.26%)	(48.27%)		(64.24%)
RED-9	2.62 (0.89)	2.68 (0.85)	1.99 (0.81)/2.96 (0.82)	2.22 (0.78)
RED-13	2.66 (0.83)	2.73 (0.79)		2.25 (0.74)
Type 2 Diabetes: n	-	19 (5.49%)		-
(%)
CoEQ: Craving for	-	44.97		-
sweet foods: M (SD)		(21.94)
CoEQ: Craving for	-	50.28		-
savory foods: M (SD)		(19.21)
Chips Consumed: M	-	-		59.41
(SD)				(39.52)

Note. Of 106 participants in the longitudinal US community sample, 94 provided follow-up data. In that sample, data for gender and age is missing for one participant. BMI and RED-9 are reported for both baseline and 6 months later. Of 165 participants in Canadian University sample, 51 completed the laboratory-based eating behavior assessment (chips consumed). Age in years; BMI = body mass index; RED-9 = Reward-based Eating Drive Scale, 9 item; CoEQ = Control over Eating Questionnaire; chips assessed in grams.

Analytic Plan

We conducted all analyses in Microsoft R open 3.5.1 (31), using 2018.08.01 version of psych, tidyverse, and cowplot packages (32–34). We drew items from the 13-item set (RED-13), which has been established as largely unidimensional, but also as having three strongly related subdimensions in previous analysis (12).

Variable preparation.

We residualized BMI and self-reported cravings for age, gender, race, education, and household income in web samples. We residualized self-reported Type 2 diabetes for age and gender, as the disease was not present in all subcategories of race, education, and income. We residualized BMI and potato chips consumed for gender in the Canadian university sample (several demographic variables were not collected in the Canadian university sample, as it was relatively homogenous).

Statistical models.

We opted for the all-subset correlation method of scale shortening due to its simplicity, as well as its alignment with our goal of developing a scale that maximally discriminates between respondents in a normal distribution (22). In other words, we intended for the newly developed short scale to maximally correlate with the full-length scale (RED-13). We determined the number of items of the RED-X by plotting the number of items on the x axis, and maximum and mean correlations within each item set with RED-13 on the y axis. For instance, there were 1287 possible combinations of a 5-item RED-X; the sum scores of all these combinations were correlated with RED-13, and the maximum and mean correlation of 5-item RED-X-s were added to the plot. The same procedure was repeated for 1–12 item versions of RED-X. After the plot was complete, we looked for the “elbow” in the emerging brevity- correlation tradeoff, the point where adding one more item would not add as much extra correlation strength as in the previous step. Because of occasional possible measurement errors when applying the instrument, we added one item to that elbow point. Once we established the desired number of items, we chose item subset that had the maximum correlation with the RED-13, and also had at least one item from each subscale, so that RED-X was conceptually as close as possible to the original RED-9 scale.

We conducted all-subset correlation analyses in parallel using Web1 and Web2 samples, and averaged the correlation values to obtain a more stable estimate. The final RED-X correlation with RED-13 was validated in the Canadian student sample to achieve an unbiased estimate of the correlation between short form and full form. We also present the correlations between RED-X and other forms of RED questionnaires across the four samples. The latter include RED-9 and RED-13 versions, which did not have RED-X items, providing an estimate that avoided correlating the same items among themselves. We assessed internal consistency with McDonald’s total omega and Cronbach alpha, as provided by “omega()” function within the psych R package (32). As alpha relies on more assumptions than omega (35), we primarily report omega total. We assessed test-retest reliability in the US community sample by using zero-order correlations between RED-9 and the short version.

We validated the final RED-X against BMI and external behaviors, such as laboratory- based eating behavior, self-report eating behavior, and diabetes status. We also compared RED- X effect sizes with RED-9 and RED-13. To mimic a typical use case, we computed Pearson correlations between the sum-scores of three versions of RED and residualized outcomes.

Results

We generated 8191 possible subsets from the RED-13 items. Figure 1 summarizes the maximum and mean correlations that RED-13 demonstrated with each subset. Depicted curves suggested that 4 or 5 items offer the best tradeoff between questionnaire length and additional explanatory power gained per extra item. We opted for 5 items rather than 4 to maximize reliability. We chose the top performing 5-item subset (henceforth, the RED-X5) from the pool of 5-item subsets that included items from each subscale (starred in Table 2). Figure 1 depicts the correlations between RED-13 and the final RED-X5 within the two web datasets, and independent replication in the Canadian dataset.

Figure 1.

Maximum and mean correlation of each RED-X subset, and final RED-X5 with RED- 13. Note. Lines denote estimates when correlations were averaged between Webl and Web2. Borders of grey ribbons denote individual values in Webl or Web2, and thus characterize variability between Webl and Web2 samples. Can=Canadian student sample, RED=Reward- based Eating Drive scale, Web1=Web1 sample, Web2=Web2 sample.

Figure 2 depicts the correlations between RED-X5 and the two other validated forms of RED (RED-9, RED-13) in all four datasets. Even when we exclude the RED-X5 items from RED-9 and RED-13, the correlations remain high in all samples, suggesting that RED-X5 captures the same construct as longer versions. RED-X5 had strong internal consistency and unidimensionality, as suggested by omega total estimates greater or equal to .80 (Figure 3). Cronbach alpha estimates had a maximum .005 absolute difference from omega total estimates. Test-retest reliability over 6 months assessed in the US community sample was r = .77 for RED- 9 and r = .72 for RED-X5.

Figure 2.

Correlations between RED-X5 and other forms of RED. Note. Can=Canadian student sample, RED=Reward-based Eating Drive scale, se=standard error, USbl = US community sample, baseline measurement, USfu = US community sample, followup measurement after 6 months. RED-13 is missing in US community sample, as RED-9 was used in that study. Web1=Web1 sample, Web2=Web2 sample.

Figure 3.

Internal consistency of different RED questionnaires. Note. Can=Canadian student sample, RED=Reward-based Eating Drive scale, USbl = US community sample, baseline measurement, USfu = US community sample, followup measurement after 6 months. RED-13 is missing in US community sample, as RED-9 was used in that study. Web1=Web1 sample, Web2=Web2 sample.

Figure 4 depicts the correlations between RED-X5, RED-9, RED-13, and external outcomes in two web samples and the Canadian sample. In all cases, the correlations between RED-X5 and the external outcomes were within standard errors of the estimates of longer versions of RED. Although most of these correlations have been documented elsewhere (12), this manuscript is the first to document an association between versions of the RED scale and laboratory food intake.

Figure 4.

Correlations between RED-X5 and outcomes. Note. All correlations significant at p<.05. See Table 1 note. BMI=Body mass index; Can=Canadian student sample, RED=Reward-based Eating Drive, T2D=self-reported type two diabetes status, W1=Web1 sample, W2=Web2 sample.

Discussion

We created RED-X5, a reliable 5-item measure of reward-related eating. We developed the RED-X5 by evaluating the length-fidelity trade-off of each possible item combination of RED-13. The final RED-X5 scale correlates well with full measures of RRE, as well as diverse external outcomes. We validated these associations in an external dataset that did not inform the item selection process. We established RED-X5 to have similar test-retest reliability over 6 months as other measures of personality and eating behaviours (36, 37). Therefore, RED-X5 is a reliable measure of RRE for use when the brevity in assessment is particularly important, such as in panel and population studies.

Our results highlight that the RRE construct associates with snacking on highly palatable food (potato chips). Previous studies have demonstrated associations between consumption of food in the laboratory and several eating-related traits (reviewed in (16)). We found that the RED-X5 was moderately associated with laboratory intake of a highly palatable food, which implicates the RRE construct as a predictor of food consumption.

Strengths and limitations.

In their seminal critique of scale shortening efforts, Smith et al (38) listed various sins of scale shortening. We believe that we avoided most of the sins as we have re-established the internal consistency, test-retest reliability and external validity of the RED-X5. The items for RED-X5 were chosen based on two datasets, and we validated the resultant scale in a third dataset that did not inform the item selection process. The RED-X5 comprises items from the original RED-9, and thus, the RED-X5 shares significant overlap with the core RED measure. However, the RED-X5 correlates with longer RED measures, even when RED-X5 items are excluded. The short version also maintained its ability to associate with various external criteria. While RED-X5 has not been tested with the eating in absence of hunger paradigm, that paradigm is known to correlate with measures capturing constructs very similar to RED (39). When questionnaire space is flexible, we recommend using the longer version of RED (RED-13), as it allows for the analysis of each subscale in explaining external behaviours. The abbreviated version described in this manuscript (RED-X5) provides only a rough estimate at the subscale level and is best used as a single score. Future research should examine the utility of RED-X5 in more diverse datasets and samples.

Taken together, we have created the RED-X5, a 5-item, psychometrically sound measure of reward-related eating. We intend for this measure to be used within large nationally representative and diverse studies, where questionnaire space is limited, so that we could learn more about RRE at the population level.

What is already known about this subject?

• Reward-related eating is a major psychological trait associated with obesity and other eating behaviors

• Reward-related eating can be measured well with Reward-based Eating Drive questionnaire (RED) which summarizes features of other popular questionnaires

• The RED questionnaire has been recently highlighted as a core measure in the ADOPT initiative (Accumulating Data to Optimally Predict Obesity Treatment).

What does current study add?

• We develop a 5-item version of RED that is suitable for large panel studies where every item counts.

• The short version correlates strongly with longer versions and is able to reproduce correlations with multiple external criteria.

• We further show (for the first time) that the RED questionnaire can predict laboratory food intake.