Predicting Eating Disorders from Internet Activity

Abstract

Objective:

Eating disorders (EDs) compromise the health and functioning of affected individuals, but it can often take them several years to acknowledge their illness and seek treatment. Early identification of individuals with EDs is a public health priority, and innovative approaches are needed for such identification and ultimate linkage with evidence-based interventions. This study examined whether Internet activity data can predict ED risk/diagnostic status, potentially informing timely interventions.

Method:

Participants were 936 women who completed a clinically validated online survey for EDs, and 231 of them (24.7%) contributed their Internet browsing history. A machine learning algorithm used key attributes from participants’ Internet activity histories to predict their ED status: clinical/subclinical ED, high risk for an ED, or no ED.

Results:

The algorithm reached an accuracy of 52.6% in predicting ED risk/diagnostic status, compared to random decision accuracy of 38.1%, a relative improvement of 38%. The most predictive Internet search history variables were the following: use of keywords related to ED symptoms and websites promoting ED content, participant age, median browsing events per day, and fraction of daily activity at noon.

Discussion:

ED risk or clinical status can be predicted via machine learning with moderate accuracy using Internet activity variables. This model, if replicated in larger samples where it demonstrates stronger predictive value, could identify populations where further assessment is merited. Future iterations could also inform tailored digital interventions, timed to be provided when target online behaviors occur, thereby potentially improving the well-being of many individuals who may otherwise remain undetected.

Within their lifetimes, about 8% of Western women and 3-3.5% of men will meet criteria for an eating disorder (ED; Galmiche et al., 2019; Raevuori, Keski-Rahkonen, & Hoek, 2014). EDs are serious mental health disorders with significant medical and psychiatric comorbidities, as well as compromised psychosocial, academic, and professional outcomes (American Psychiatric Association, 2013; Galmiche et al., 2019; Raevuori, Keski-Rahkonen, & Hoek, 2014). The insidious nature of the disorders, mental health stigma, and complex cultural messages about appearance, eating habits, and thinness all hinder early identification and intervention (Ali et al., 2017). Patients report it took them, on average, 3.6 years to acknowledge their ED and seek specialized intervention (de la Rie, Noordenbos, Donker, & van Furth, 2006). Further, in several recent studies, 58.5% to 85.9% of respondents meeting criteria for an ED reported never being in treatment (Fitzsimmons-Craft, Balantekin, Eichen, et al., 2019; Kazdin, Fitzsimmons-Craft, & Wilfley, 2017; Sadeh-Sharvit et al., 2018). Left untreated, EDs can become chronic and harder to improve (Wonderlich et al., 2012).

Given the significant personal and public health costs associated with EDs and the barriers to seeking treatment, there is an urgent need to explore innovative ways to identify cases or individuals at high risk for EDs and offer them accessible interventions (Striegel Weissman & Rosselli, 2017; Taylor et al.). Technology provides innovative methods to identify individuals who may be at risk for a mental disorder, assess the severity of their symptoms, and tailor personalized interventions. Passively collected data - such as mobile app use patterns, number of steps a day, and location – can all provide valuable information about a person’s mental condition (Firth et al., 2018; Lazar et al., 2017). These data can be gathered with little participant burden and from individuals who may not have completed an ED screening otherwise (Ware et al., 2018), thereby increasing the ability to detect a wide range of individuals with EDs. The Internet is a ubiquitous platform which is accessed daily by 90% of U.S. adults (Anderson, Perrin, Jiang, & Kumar, 2019). Internet activity (e.g., web browsing history, usage patterns, and search engine queries) has been shown to be a feasible method in predicting medical problems such as pancreatic cancer, Celiac disease, and diabetes (Hochberg, Daoud, Shehadeh, & Yom-Tov, 2019; Lebwohl & Yom-Tov, 2019; Paparrizos, White, & Horvitz, 2016). There has been some research recently related to using online activity to identify individuals with mental health disorders. For example, studies demonstrated that patients with depression tended to have more Facebook posts than patients without depression, but there was no significant difference in the content of the posts they had authored (Smith et al., 2017). Another study demonstrated that depression and post-traumatic stress disorder can be identified in Twitter users, based on the content of their posts (Reece et al., 2017). The ability to identify individuals with various problems also has the potential to be used to ultimately provide access to various resources, including digital interventions (Taylor et al., 2019). To our knowledge, browsing history has not been used to identify EDs.

Due to the need to develop novel means for identifying EDs, the goal of the present study was to investigate whether Internet activity can predict the presence of an ED or high risk for this illness. In particular, this research aimed to explore whether data from a clinically-validated online screen could be predicted by a signal from Internet activity, to inform future identification and early intervention efforts to reduce the incidence and prevalence of EDs. The study is unique in attempting to identify both those at risk of or with an ED.

Methods

Participants and procedure

Participants were recruited from the crowdsourcing website Prolific Academic (prolific.co). They were invited to partake in a study about eating habits. Inclusion criteria were being female, 18 years of age or older, and use of a desktop computer running the Windows operating system. We chose to focus on females as ED presentation differs between the genders (Murray et al., 2017). This study was approved by the University’s Institutional Review Board.

Following the consent and onboarding process, participants were asked to complete the Stanford-Washington University Eating Disorder Screen (SWED; Graham et al., 2019), an empirically-supported online assessment of ED risk or clinical status. This questionnaire categorizes the respondents into several risk/diagnostic groups: (a) a clinical/subclinical ED, including the subclassifications of possible anorexia nervosa (AN), bulimia nervosa (BN), and binge-eating disorder (BED) and Unspecified feeding or eating disorder (UFED); (b) high risk for an ED (characterized by elevated weight and shape concerns); (c) no ED; or (d) probable Avoidant/Restrictive Food intake disorder (ARFID) (Graham et al., 2019). After completion of SWED, participants were asked (but not required) to contribute their Internet browsing activity. Participants were informed that these data would only be used for the purposes of studying whether EDs can be detected from browsing histories. Participants were further advised that data would be used in this research project only, and that they would receive the same payment compensation, regardless of whether they contributed their browsing history or not. If they agreed, participants downloaded and ran a software program on their computer, which collected browsing history from any browser installed on their computer into a file, and subsequently uploaded this file to the study website. This method of collecting browsing history has been successfully used in past research (Schueller, Steakley-Freeman, Mohr, & Yom-Tov, 2020). Finally, participants were asked whether they would be interested in receiving feedback on their SWED score. Those choosing feedback were provided a brief summary of the results of the screen, and a link to resources from the National Eating Disorders Association (Fitzsimmons-Craft, Balantekin, Graham, et al., 2019).

Data analysis

Participant willingness to contribute data was analyzed using an Analysis of Variance (ANOVA) with pairwise interactions, where the dependent variable was whether the user uploaded any browsing history data and the independent variables were age, current body mass index (BMI), and ED type.

The browsing history of no more than the last 6 months was used in the data analysis, to capture participant online behaviors during the time period while they may have had an ED. Each participant was represented through several attributes derived from their browsing history (Hochberg et al., 2019, Schueller et al., 2020). We employed feature selection in conjunction with model generation, a strategy known as “wrapper” feature selection (Blum and Langley, 1997). It allows defining a larger number of possibly relevant attributes, from which the most informative for improving classification accuracy is selected. These attributes were chosen in advance to represent activity related to EDs, through specific keywords mentioned in the data, possible demographic differences (activity on specific social networks and on other internet platforms, and demographic attributes), and attributes quantifying general activity online (Backstrom, 2009, Yom-Tov et al., 2014), including temporal pattern of activity. Thus, the computed features represent attributes that are derived from past work on predicting medical outcomes from internet data (Hochberg et al., 2020, Schueller et al., 2020), attributes representing demographic differences (use of Bing vs. Google, country specific websites, age, activity on social networks), and attributes quantifying general activity patterns online (diurnal and weekly activity patterns, activity on social networks. We note that these attributes were derived directly from internet data or are available to internet providers. This includes demographic details such as age and gender which are commonly provided at the time of registration to an internet service or can be predicted from internet use (Goel, Hofman, & Sirer, 2012). Age and sex were reported by participants in this study directly, and not assumed based on participants’ Internet activity. We note that we employed feature selection (see below) for the machine learning models used in this study. Hence, it is preferred to include a large number of model input attributes which could be relevant and allow the model to select the most predictive ones (Duda, Hart, & Stork, 2012). The attributes extracted included:

1. Use of specific terms that could potentially be related to EDs and to mental health in the title or the name of the website as shown in Table 1. The keywords were identified in two ways: First, each of the authors proposed keywords in each of the topics represented in the Table. Second, the 200 most common words and word pairs in the title of the pages of the training data were examined and those were categorized by the authors into the topics in Table 1, if those words were relevant to EDs.

2. Activity on social networks: Average number of daily visits to social networks, stratified as follows:

   1. Popular social networks, including: Pinterest, Facebook, Reddit, LinkedIn, Instagram, Twitter, and Tumblr.
   2. Private social networks: Those with the title forum, community, or the website Omegle.
   3. Dating social networks, including: okcupid, plentyoffish, match, adultfriendfinder, fetlife, or adam4adam
   4. Video websites, including: YouTube, Vimeo, and Metacafe.
   5. Gaming websites, including: video game, pcgamesn, humble bumble, apex legends, captain marvel, steam, discord.

3. Fraction of searches made on Google compared to those made on Bing.

4. Fraction of country-specific websites.

5. Activity rates per day: Average and median browsing events per day and the average number of active hours per day.

6. Average hourly and weekday activity.

7. Age of participants in years. This variable was used as many Internet providers have access to such demographic data.

Table 1.

Browsing keywords and their categories

Category	Keywords
Mental Health	survivor, depress, suicid, anxiety, anxious, self-image
Treatment	psycholog, psychiat, treatment
Eating disorder	proana, pro-ana, thinspo, anorexia, bulimia, eating disorder, promia, thinspiration, vomit, laxative, diuretic, water pill, fasting, binge
Diet	diet, skinny, scale, measure, loss, binge, shape, weight, fit, calories, BMI, workout, gym, thin, shape, exercise, body
Body	shapeless, fat, unattractive, cellulite, cosmetic, shampoo, hair, skin, salon, beaut, stomach, belly, thigh, hips, boob, breast, butt, bottom, makeup
Imagery	jpeg, jpg, photo, image, video, gfycat, imgur
Food	breakfast, lunch, dinner, meal, organic, eating, , food, recipe
Sex	cheating, tinder, sex, masturbate
Clothing	boohoo, etsy, asos, urbanoutfitters
Allergy	gluten, celiac, allergy, allergies, food sensitivit, vegan, vegetarian, paleo, low carb, lowcarb
Stimulant drugs	amphetamine, dextroamphetamine, methamphetamine, dextroamphetamine, levoamphetamine, lisdexamfetamine, methamphetamine, adderall, dexedrine, procentra, dextrostat, ritalin, concerta, vyvanse, focalin, strattera
Diet pills	orlistat, xenical, belviq, lorcaserin, cetilistat, sibutramine, reductil, meridia, rimonabant, acomplia, zimulti, metformin, glucophage, exenatide, liraglutide, semaglutide, byetta, amylin, pramlinatide, phentermine, topiramate, bupropion, naltrexone, redux, fen-phen, pyruvate, metamucil, citrucel, fibercon, docusate, colace, diocto, gibs-eze, glycerin, hallens, sorbitol, lactulose, peg, colyte, miralax, epsom salt, magnesia, senna, bisacody, castor oil, cascara, buckthorn, bisacodyl, lasix, furosemide, torasemide, demadex, tortas, wator, thiazide, diuril, acetazolamide, diamox, neptazane, methazolamide, spironolactone, aldactone, aldosterone, amiloride, midamor, triamterene, dyrenium, mannitol, adipex-p, fastin, diethylpropion, tenuate, green tea extract, hydroxycitric, cla, hoodia, chitosan, aloe, glucomannan, guarana, yerba, guar gum, ephedra, ma huang, adipex, phenteramine, topamax, desoxyn, alli, qsymia, saxenda, hcg, pregnyl, tagamet, bontril, topiragen

Screen results were grouped into 3 ED groups: (1) No ED or risk for ARFID; (2) High-risk for an ED; and (3) Clinical/subclinical AN, BN, BED, or UFED. The disorders in the clinical group reflect increased weight and shape concerns and an internalization of the thin ideal (Fairburn, 2008), and individuals with BN, BED, and UFED have been included in studies testing online interventions for EDs (Fitzsimmons-Craft, Firebaugh, et al., 2019; Fitzsimmons-Craft et al., revision invited). In contrast, an ARFID diagnosis excludes body image disturbances (American Psychiatric Association, 2013). Therefore, respondents with no ED or ARFID were grouped in the current study, and ARFID was not included in the clinical/subclinical ED group. Next, we attempted to predict membership in these three groups. We used a multiclass predictor based on a reduction to a two-class model with error correcting output codes (Dietterich & Bakiri, 1994). The two-class predictors were GentleBoost (Friedman, Hastie, & Tibshirani, 2000), which are known to provide high accuracies, with 100 trees (Fernández-Delgado, Cernadas, Barro, & Amorim, 2014). Results for other models, including decision trees, support vector machine (SVM, with error correcting output codes for multiclass classification), and logistic regression are also provided. Our performance criterion was accuracy as measured by the percentage of correctly classified ED group. To reduce the likelihood of over-fitting, we estimated accuracy using leave-one-out cross-validation (Duda, Hart, & Stork, 2012), i.e. we trained a model to predict outcomes for all attributes and then use the most predictive subset to predict the outcome for that sample, repeating the process for all samples. Attributes were selected using sequential forward feature selection (Duda et al., 2012), i.e., selecting the best single attribute, and then sequentially adding the most predictive attribute. The attributes were selected on a random sample of 50% of the data. These attributes were then tested on the remaining 50% of the data. Lastly, we used a table of confusion to depict the differences between the predicted ED group and observed numbers (Jacobson, Weingarden, & Wilhelm, 2019). All analysis was performed using Matlab 2019a.

Results

Sample characteristics

A total of 936 females were recruited for this study. Average age of contributing participants was 26 years old (SD = 5, range: 18-61) and the average BMI was 25 (SD = 5, range 16-40). Of those, 231 (24.7%) contributed their browsing history with at least 10 days of data (average length among those contributing: 273 days, range: 10-3517 days). The distribution of ED types as determined by SWED and the percentage of participants of the total sample and of each ED group contributing their search histories appears in Table 2. Twenty-five percent of participants were from the UK, 24% from the US, 5% from Canada. Additionally, 42% were from other European countries and the remaining 4% from other countries or unknown. Sixty-eight percent of the sample reported working full or part time.

Table 2.

Distribution of eating disorders among study participants (N = 936).

Eating disorder classification	Percent of sample (N = 936)	Participants in this group that contributed their search histories (percentage out of total sample)	Participants in this group that contributed their search histories (percentage out of their respective eating disorder group)
No ED	48%	20%	9%
ARFID	16%	30%	42%
Clinical/Subclinical BN or BED	17%	27%	35%
High risk ED	17%	19%	25%
AN	2%	3%	33%
UFED	<1%	1%	75%

Note: ED = Eating Disorder; ARFID = Avoidant/Restrictive Food Intake Disorder; BN = Bulimia Nervosa; BED = Binge Eating Disorder; AN = Anorexia Nervosa; UFED = Unspecified Feeding and Eating Disorder.

ANOVA found that none of the variables (i.e., age, current BMI, ED type, country, employment status) were significantly related (p > 0.05, with Bonferroni corrections) to participants’ willingness to provide browsing history. This is in line with previous work (Schueller et al., 2020), which also did not find such correlations.

Participant Internet activity

Feature selection revealed that the best accuracy to predict ED group in the sample was obtained by using the following 4 attributes concurrently:

1. Use of keywords related to EDs (see Table 1).

2. Participant age.

3. Median browsing events per day.

4. Fraction of activity at noon (12pm).

Table 3 is a ‘confusion’ table presenting percent of observed and predicted percentages of participants in each group according to the machine learning algorithm. As the table shows, the total accuracy reached by the algorithm (on the testing set) was 52.6%, compared to a baseline (random decision) of 38.1%, a relative improvement of 38%. However, the model classified very few participants as having high risk, but instead grouping them into the No ED or ARFID group. Table 4 presents the percentage of participants from each group who mentioned keywords in each of the categories, and who were active on each of the social network categories. Figure 1 shows the diurnal activity patterns by ED group. Participants in the high risk for ED group were more active during the morning hours, while those in the clinical/subclinical ED group were more active in the early afternoon.

Table 3.

A table of confusion^a for the percentage (N) predicted versus measured number of participants in each ED group (results are shown in percentage of actual cases of participants contributing their Internet activities histories, N = 116).

		Predicted ED group
		No ED / ARFID	High risk for ED	Clinical/Subclinical ED
Observed ED group	None / ARFID	65 (36)	5 (3)	29 (16)
	High risk for ED	32 (6)	37 (7)	32 (6)
	Clinical/Subclinical ED	43 (18)	14 (6)	43 (18)

^aIn machine learning, a table of confusion (also known as a confusion matrix or an error matrix) presents the correct classifications according to the model compared with the computed algorithm

Table 4.

Percent of respondents in each ED group who used keywords in each of the keyword categories and who were active on each social network category.

Percent mentioning	No ED / ARFID	High risk ED	Clinical/Subclinical ED
Keywords
Mental Health	63.7	65.9	66.2
Treatment	57.5	63.6	63.5
Eating disorder	15.0	13.6	36.5
Diet	97.3	97.7	97.3
Body	87.6	90.9	86.5
Imagery	99.1	100.0	98.6
Food	96.5	90.9	91.9
Sex	80.5	75.0	82.4
Clothing	48.7	47.7	50.0
Allergy	54.9	63.6	59.5
Stimulant drugs	5.3	4.5	12.2
Social network category
Popular social networks	98.2	97.7	100.0
Private social networks	93.8	97.7	90.5
Dating	92.9	95.4	91.9
Video	99.1	100.0	100.0
Gaming	64.6	77.3	62.2

Figure 1.

Diurnal activity pattern by ED group.

Other prediction models

A logistic regression model with one-versus-all multiclass-to-two-class reduction was applied solely to the keywords (see Table 1) and reached an accuracy of 44.0% (compared to 52.6% using a random forest). Thus, the additional attributes improved accuracy over using only keywords related to EDs. Further, a random forest model applied to age alone reached an accuracy of 36.2%, comparable to random decision.

As noted in the Methods, we attempted to use other classification models to predict ED group. However, lower classification accuracies were reached. Specifically, decision trees reached an accuracy of 37.1%, SVM 46.7%, and logistic regression 37.9%. However, SVM and logistic regression did not classify any of the test set participants as being in the high risk ED category.

Discussion

Predictors for most mental health disorders that can be leveraged to actionable intervention efforts are rare (Goldman, 2019). A significant improvement of existing models for early identification of most diseases, including EDs, could potentially reduce personal and public health costs, and hence is a society-level priority. The current study demonstrated that a machine learning algorithm can significantly identify women screening positive for EDs using a combination of a number of variables: age, search of online information about ED symptoms and ED-promoting websites, proportion of Internet activity per hour, and amount of activity at noon. Findings suggest that a preliminary screening tool that explores online activity history may increase prediction of whether an individual has an ED. Further, this study found that respondents meeting criteria for an ED engage with more online materials and websites related to EDs and search more for information about stimulant drugs (both specific and non-specific to attention deficit/hyperactivity disorder), substances whose side effects include reduced appetite, and are abused by individuals with EDs for weight loss purposes (EL et al., 2016). Participants in the ED group also differed from their counterparts by being more active online in the early afternoon.

Findings of this study suggest that Internet browsing behavior can be used to screen for EDs, using a novel method of non-disruptive data, collected in the individual’s natural environment. Methodologically, our results suggest that a quarter of our sample was willing to contribute their Internet data for a study of mental health conditions. Moreover, we did not identify any predictors of unwillingness to contribute data. Thus, the proposed methodology represents a cost-effective method for data collection from relatively large populations. Future studies will aim to identify whether consistent bias exists in such willingness to participate, and whether different prompts can encourage greater participation. Should these findings be replicated in future studies and with new cases of EDs, they may bear significant public health implications. An early signal of probable ED psychopathology could be leveraged to inform individuals of their risk for having the disorder and to provide psychoeducation and referrals to treatment resources (Fitzsimmons-Craft, Balantekin, Graham, et al., 2019). For instance, a short screening survey can provide useful data to those who are already aware that their eating-related cognitions and behaviors may be pathological. However, for individuals in the pre-contemplation phase, an algorithm identifying specific Internet activity could trigger search engine ads providing psycho-education and referrals or a link to complete an online ED screen (Yom-Tov, Brunstein-Klomek, Mandel, Hadas, & Fennig, 2018, Yom-Tov, 2020). Notwithstanding the fundamental commitment to maintain patient privacy and confidentiality, future studies should test whether Internet activity in patients with EDs is associated with data from their electronic medical record and apps they use (Asch et al., 2019; Tregarthen et al., 2020). Data on social media activity in identified populations should also be integrated in future models to predict risk factors and early symptoms (Cavazos-Rehg et al., 2019; Taylor et al., 2019), which we were not successful in doing in this study. Further, research to explore whether targeted content for individuals in high risk is engaging and effective is also warranted (Taylor et al., 2019). From a public health standpoint, analyzing big data from search engine queries could inform lawmakers on the needs of the wide populations, and promote timely response to increases in ED symptomatology (Ginsberg et al., 2009).

This pilot study has some limitations. Although original sample size was large, only a quarter of participants contributed their data, limiting the external validity of findings. Future studies will test how sample size affects model accuracy. A larger study could also test the stability and efficacy of the algorithm developed in this investigation to populations with a lower base rate of eating problems. Further, although the prediction accuracy was significantly improved over the baseline, it requires additional revision before it can be used in real-life setting. Algorithm accuracy was limited due to dataset size, collection of desktop data (versus mobile devices), and the difficulty of the task, which required differentiating among similar human conditions, some of which are not clearly reflected in Internet behavior. A larger sample could also allow more detailed representations of the data, for example, removing the need to group keywords by topic and instead representing each keyword as an attribute. Further, while this study recruited female participants, EDs affect all genders, ages, races, and ethnicities, thereby strengthening the case for recruiting diverse samples in a larger replication study. Additionally, our data comprised of desktop browsing histories, not smartphone or tablet data. Since evidence suggests that sensitive topics are often browsed on the latter platforms (Pelleg, Savenkov, & Agichtein, 2013), it may be that using these data would enhance our prediction ability. However, currently, technical challenges preclude this data collection. We note that our results suggest that identifying participants in the high risk for an ED group, rather than an ED or no ED groups, is difficult. Instead, it may be necessary to focus on differentiating individuals with fewer versus greater symptoms. Finally, since a quarter of participants were willing to provide their data, different prompts and explanations about the reason for collecting Internet activity histories could be tested in future investigations.

Recent technological advancements offer innovative ways for feasible and effective screening and early intervention for individuals with mental health disorders (Sadeh-Sharvit, 2019). The predictive accuracy of the model studied here was moderate, and future investigations should evaluate whether Internet browsing history could be clinically useful in detecting ED status, particularly in light of ethical and privacy considerations. Should a predictive model be implemented in the field of EDs, it has the potential of reducing the costs associated with delayed identification of the disorder and its medical comorbidities. Given the high rates of Internet adoption, browsing histories and online activity patterns could be harnessed to ultimately reduce the incidence and prevalence of EDs, just as they are now harnessed for targeted marketing.