Analysis of the Accuracy of Weight Loss Information Search Engine Results on the Internet

Abstract

Objectives. We systematically identified and evaluated the quality and comprehensiveness of online information related to weight loss that users were likely to access.

Methods. We evaluated the content quality, accessibility of the information, and author credentials for Web sites in 2012 that were identified from weight loss specific queries that we generated. We scored the content with respect to available evidence-based guidelines for weight loss.

Results. One hundred three Web sites met our eligibility criteria (21 commercial, 52 news/media, 7 blogs, 14 medical, government, or university, and 9 unclassified sites). The mean content quality score was 3.75 (range = 0–16; SD = 2.48). Approximately 5% (4.85%) of the sites scored greater than 8 (of 12) on nutrition, physical activity, and behavior. Content quality score varied significantly by type of Web site; the medical, government, or university sites (mean = 4.82, SD = 2.27) and blogs (mean = 6.33, SD = 1.99) had the highest scores. Commercial (mean = 2.37, SD = 2.60) or news/media sites (mean = 3.52, SD = 2.31) had the lowest scores (analysis of variance P < .005).

Conclusions. The weight loss information that people were likely to access online was often of substandard quality because most comprehensive and quality Web sites ranked too low in search results.

Despite the Surgeon General’s call to action to address the obesity epidemic in 2001,¹ the prevalence of obesity in the United States surged dramatically from less than 15% in 1990 to 36% in 2010.^2,3 The impact of obesity has recently led the American Medical Association to officially label it as a disease.⁴ Although there are diverging data on the actual number of deaths related to obesity,⁵ one estimate is that approximately 300 000 deaths are directly attributable to obesity every year in the United States,^6,7 and that obesity-related comorbidities, (type 2 diabetes, hypertension, or osteoarthritis) significantly impact the quality of life of patients.⁷ Excess weight results in 95 million years of life lost⁸ and a staggering $147 billion in medical care costs in the United States.⁹

National data suggest that 64% of Americans are trying to lose weight and that 48% are following a weight loss strategy.¹⁰ Patients are increasingly turning to the Internet for information about health in general, and for weight loss in particular. Currently, 82% of Americans use the Internet, and 66% have high-speed Internet at home¹¹; nearly 60% seek health information on the Internet,¹² and 18% do so on their mobile devices.¹³ According to national data,¹⁴ 42.83% of Internet users access the Web for weight loss and physical activity information. Factors associated with Internet use for weight loss and physical activity information are age, education, being married, African American or Hispanic race/ethnicity, and higher body mass index (BMI; defined as weight in kilograms divided by the square of height in meters).¹⁴ Studies show a correlation between Internet use and health behavior.^15,16

One of the main reasons for the increase in Internet use for health information over the past 10 years is the great improvement in search engine technologies that has resulted in easy access to an enormous amount of information (between 200 and 500 terabytes). Because people turn mostly to general search engines to find health information,¹² rather than the government-sponsored, health-related sites such as pubmed.org, myplate.gov, etc., the accuracy of health information accessed is unclear. Currently, search engines that are not health specific do not have the capability to assess the quality of the content of search results. For instance, Google (Menlo Park, CA) ranks pages primarily (although not exclusively) through its PageRank algorithm (US Patent 6,285,999, publication date September 4, 2001), which represents a recursive count of pages linking to a given page. In essence, PageRank is a measure of online popularity, but it is subject to manipulation through Web page design. Studies evaluating Web site quality and content have focused on the user or consumer perspective of the Web sites. WebQual¹⁷ is the primary evaluation tool. It is based on the theory of reasoned action and the technology acceptance model, and is largely validated from a consumer perspective (e.g., marketing).¹⁷ Analytical hierarchical processes have also been used to evaluate Web site quality. For instance, fuzzy analytical hierarchical processes are used to rate online courses,¹⁸ or e-banking,¹⁹ and focus mostly on the usability aspects of Web sites from a user's point of view.

Although evaluation of the content accuracy of health information has been done in a very ad hoc manner, we do not have a measure of the accuracy of the information obtained by Internet users performing online searches. This is a concern from both a health care provider and public health perspective. Scullard et al.²⁰ evaluated the information accuracy of Google searches pertaining to children’s health. Abbott²¹ and Kata²² evaluated vaccination misinformation on the Web. However, to our knowledge, no tool exists to evaluate the quality of online information pertaining to weight loss.

To curb the obesity epidemic, authoritative evidence-based guidelines have been published for the management of overweight and obesity in adults.²³ They recommend health care providers identify patients who need to lose weight, and that they match treatment benefits with risk profiles. For diet, they recommend 1200 to 1500 kilocalories per day for women and 1500 to 1800 kilocalories per day for men, or a 500 to 750 kilocalories per day energy deficit, or restriction of certain foods (e.g., high carbohydrate foods, low-fiber foods, or high-fat foods). They recommend that obese individuals participate in a comprehensive lifestyle program for at least 6 months. For weight loss maintenance, they recommend high levels of physical activity (i.e., 200–300 minutes/week) and continued dietary changes. Selected patients (BMI ≥ 40 or ≥ 35 kg/m² with obesity-related comorbid conditions who have not responded to behavioral treatment with or without pharmacotherapy) may benefit from bariatric surgery. Pharmacotherapy is recommended at the time of initiation of a lifestyle intervention program on selected patients who have been unable to lose weight or sustain weight loss.²³ Despite extensive research on the use of pharmacotherapy alone or in combination with nutrition and physical activity for weight loss,^24–27 and the large amount of nutritional supplements that claim to help with weight reduction, currently, the Food and Drug Administration has only approved a few medications for this purpose: orlistat (Alli, GlaxoSmithKline, London, UK), lorcaserin (Belviq, Arena Pharmaceuticals, San Diego, CA), and phentermine in combination with topirimate (Qsymia, Vivus Inc., Mountain View, CA).

Our goal was to evaluate both the quality and comprehensiveness of the content that Internet users performing online searches related to weight loss and weight management are likely to read.

METHODS

Algorithmic differences to retrieve, archive, and index information make the results of an online query search engine–dependent. Therefore, we had to decide which search engine to use in this study. StatCounter (StatCounter, Dublin, Ireland) is a free online tool that provides statistics pertaining to search engine and browser (which does not affect the results of online searches) usage, and offers results globally or by region, and by viewing mode (mobile or desktop). The statistics are based on aggregated, quality-checked data provided by 15 billion page views for more than 3 million Web sites, and are updated every 4 hours. According to this data, Chrome (Google) has been the most used browser since May 2012²⁸ (38%), compared with Internet Explorer (Microsoft, Redmond, WA; 29%), or Firefox (Mozilla Inc., Mountain View CA; 20%). Google is the most frequently used search engine²⁹ (91.5%) ahead of Bing (Microsoft; 8%). Finally, as of June 2012, more than 85% of searches were performed on a desktop, and fewer than 15% were performed on a mobile device.³⁰ Therefore, we performed all our queries using Google on the Chrome browser.

Identification of Web Sites for Evaluation

We generated a series of search terms (queries) as the initial step in identifying Web sites for inclusion in our study. First, we approached a convenience sample of 20 individuals known to the authors: 15 were interested in losing weight (9 women, 11 men; 15 Hispanic, 5 White; mean age = 39 [SD = 9.1] years; 12 had a high school diploma and some college, 8 had a college degree). We asked what terms or sentences they would use to find information on weight loss, in an Internet search. Second, we selected additional keywords (weight loss, lose weight, diet) and used the autocomplete³¹ of Google to generate the first set of queries. Finally, we complemented our queries using google.com/trends, which is geospecific to the United States, to identify the most common terms pertaining to weight loss in the United States at the end of 2012. Between November 1 and November 7, 2012, we produced a set of 30 unique queries after elimination of duplicates. Each query generated between 250 000 and 500 000 000 results.

The next step was to develop a protocol to identify the links that people were most likely to view. Although it was not possible to obtain actual data directly from Google on which links were viewed in a search engine result page (SERP) because these data were not shared by the company, data from other sources (e.g., search clicks; the links that were clicked on)³² showed that for predefined search tasks, the first 5 links collected more than 90% of the clicks. This was corroborated by several informal studies done by marketing firms (search engine optimizers), which showed that 96% of the clicks were on the first SERP, and 88% of the clicks were on the first 5 links.^33,34 Google also has sponsored ads that allow companies to pay to be at the top of searches. Therefore, for the 30 identified queries, we selected all the sponsored ads and the first 5 links resulting from each query. Exclusion criteria were Internet protocol–specific Web sites (e.g., search results that were associated with the location of where the search was performed), and pages that were purely commercial (i.e., only provided products to purchase, without information). Because most pages contained numerous links, we focused only on the main page, unless the query resulted in an article spread over multiple pages, or if the article contained a video that supported the main page. This decision was consistent with the close to linear eye pattern used in online searches,³² and it ensured consistency when we evaluated different Web sites that had different ways to display information. Finally, we excluded Internet forums from our results because forums first pages usually consisted of a list of links.

Web Site Classification and Evaluation

Web sites were first separated (Figure 1) into sponsored ads and nonsponsored ads. Sponsored ads were easily identified in a Google search because they were identified as such and highlighted in light yellow. They were all categorized as commercial Web sites. The remaining nonsponsored SERPs were categorized as commercial sites; media or online media sites (informational); blogs; medical, government, or university sites; and others. This nomenclature is standard for Web sites, aside from medical, government, or university sites, which were added, given the purpose of our research. We considered commercial all pages that provided only products for sale, without additional information. Media or online media sites were identified as multiauthor publications in the general press. Blogs were sites with reverse chronological order, discrete entries, usually by a single author or small group of authors, that allowed interaction from their readers through the comment section. Medical, university, or government sites were identified as either written by a doctor, a health care professional, or a group of doctors and health care professionals, or sites with a .gov or .edu extension. Finally, several of the SERPs did not fit in any of these categories, were heterogeneous in nature, and were classified as other (e.g., nonprofit organizations, wiki pages).

FIGURE 1—

Inclusion or exclusion and Web sites classification: Accuracy of Weight Loss Information Search Engine Results on the Internet, 2012.

Note. med/uni/gov = medical, university, or government sites.

To develop a quality metric for each Web page, we adapted 2 previously defined approaches.^21,35 We assessed 3 aspects of each Web site and developed scoring criteria for each: (1) accuracy of information content related to weight loss, (2) author credentials, and (3) Web site design features.

We developed the content scoring criteria (Figure 2) based on evidence-based literature for weight loss available at the time of the review.^{25–27,38,44–46} We evaluated quality separately for each of the 5 categories of information: with score ranges of 0 (nothing) to 4 (excellent) for nutrition, physical activity, and behavioral changes, and score ranges of 0 (nothing) to 2 (excellent) for pharmacotherapy and surgical options, with an overall maximal score of 16. Because the pharmacotherapy and surgery options were done only in a medical setting and in conjunction with changes in nutrition, physical activity, or behavioral changes, we scored the 3 latter strategies higher, and we looked specifically at the sites subscores across these 3 approaches. To evaluate the comprehensiveness of a Web site, we scored each between 0 and 3, depending on the number of key categories (nutrition, physical activity, and behavioral changes) covered.

FIGURE 2—

Rubric evaluation form.

The authorship scoring criteria included whether the author was identifiable, whether they had a degree in a related field or organization, and whether contact information was available (6-point scale). Finally, the scoring criteria pertaining to design (maximum 6 points) focused on the overall form of the Web page (structure, minimal page layering, and minimal ads). We also evaluated reading level by using http://www.read-able.com and recording the reading level. This Web site uses an established scoring mechanism to evaluate reading levels (e.g., Flesch–Kincaid) based on a 0 to 100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.⁴⁷

In addition, we recorded whether references were provided, hyperlinked, and reputable, that is, published in a peer-reviewed journal or by an authoritative source. We noted when the Web site was last updated. We also noted whether unsubstantiated claims were made, and defined claims as unsubstantiated if they were not included in any of the evidence-based recommendations.^{25–27,38,44–46} Unsubstantiated claims did not affect the scoring.

Procedure and Analysis

One of the authors (F. M.) with computer science expertise developed the algorithm for the search terms and performed the searches in November 2012. We checked the Web pages for inclusion and exclusion criteria, and then shared them with the whole team. Three investigators (F. M., N. K. S., E. P.) independently scored each Web site from February to March 2013. The scores were aggregated across investigators to obtain subscale and total scores. For the analysis, we utilized descriptive statistics to describe the scores, and examined the scores by type of Web site, using a 2-sided test of significance with analysis of variance.

RESULTS

The 30 selected queries (Table A, available as a supplement to the online version of this article at http://www.ajph.org) resulted in 129 unique sites (Figure 1). We excluded results that were specific to the Internet protocol address of the computer used (n = 1), Internet forums (n = 10), videos (n = 3), advertising pages (n = 7), and pages that were removed from the Internet (n = 6). Of the remaining 108 pages, 17 were sponsored ads and 91 were nonsponsored ads. However, our final analysis was limited to 103 pages since the time when the pages were analyzed (February–March 2013) because 4 sponsored ads and 1 nonsponsored page had been taken off the Internet. The remaining 103 Web sites were composed of 21 commercial sites (13 sponsored ads and 8 nonsponsored commercial sites), 52 news or online media sites, 7 blogs, 2 government or university sites, 12 medical sites, and 9 unclassified sites (charities, nonprofit organizations, etc.) We grouped sponsored ads and nonsponsored commercial sites because these sites have the same purpose, and their evaluations were very similar during the analysis of the scoring.

Mean comprehensiveness was 2.43 (SD = 0.82), and mean comprehensiveness per Web site type was 1.95 (SD = 1.12) for commercial sites, 2.42 (SD = 0.75) for news or online media sites, 3.00 (SD = 0.00) for blogs, 2.71 (SD = 0.61) for government or university sites, and 2.78 (SD = 0.44) for unclassified sites.

The content quality score for nutrition was mean = 1.57 (SD = 0.99); for physical activity, the score was mean = 1.08 (SD = 1.01); behavioral changes were a mean = 0.95 (SD = 0.91); pharmacotherapy was a mean = 0.11 (SD = 0.39); and surgery was a mean = 0.04 (SD = 0.21).

The content scores pertaining to the subset that consisted of nutritional, behavioral, and physical activity (maximal score of 12, results not shown) were also calculated; however, only 4.85% of Web sites attained a score of 8 or higher, and only 23% of pages scored 6 or higher. Nutrition subscores were higher than other content subscores (Table 1). Very few Web sites mentioned anything about surgery options (n = 4) or pharmacotherapy (n = 14).

TABLE 1—

Mean Subscores and Total Score by Type of Web Site: Accuracy of Weight Loss Information Search Engine Results on the Internet, 2012

	Content Subscores
	Nutrition (Range = 0–4)	Physical Activity (Range = 0–4)	Behavioral (Range = 0–4)	Pharmaco-therapy (Range = 0–2)	Surgery (Range = 0–2)	Authorship (Range = 0–6)	Content (Range = 0–16)	Design (Range = 0–6)	Total Score (Range = 0–28)
All sites (n = 103), mean SD	1.57 (0.99)	1.08 (1.01)	0.95 (0.91)	0.11 (0.39)	0.04 (0.21)	0.82 (0.96)	3.75 (2.48)	3.62 (0.74)	8.07 (3.40)
Commercial (n = 21), mean SD	1.09 (1.05)	0.71 (1.12)	0.45 (0.72)	0.11 (0.32)	0.10 (0.37)	0.38 (0.80)	2.37 (2.60)	3.37 (0.89)	5.92 (3.42)
News/media (n = 52), mean SD	1.52 (0.91)	0.99 (0.91)	0.96 (0.98)	0.05 (0.28)	0 (0)	0.65 (0.79)	3.52 (2.31)	3.48 (0.72)	7.63 (3.13)
Blog (n = 7), mean SD	2.52 (0.57)	2.05 (1.01)	1.67 (0.86)	0.10 (0.16)	0 (0)	0.43 (0.79)	6.33 (1.99)	4.05 (0.23)	10.81 (2.35)
Med/uni/gov (n = 14), mean SD	1.86 (1.11)	1.18 (1.05)	1.35 (0.71)	0.42 (0.75)	0.02 (0.09)	2.07 (0.92)	4.82 (2.27)	4.27 (0.27)	10.67 (3.17)
Unclassified (n = 9), mean SD	2.07 (0.28)	1.52 (0.73)	0.93 (0.62)	0 (0)	0.15 (0.44)	1.11 (0.78)	4.67 (1.33)	3.70 (0.63)	9.48 (1.25)
Analysis of variance P	.001	.018	.008	.027	.213	< .001	< .001	.001	< .001

Note. med/uni/gov = medical, university, or government sites.

Mean content quality score (Table 1 and Figure 3) of a possible 16 points was 3.75 (SD = 2.48). The content quality score varied significantly by type of Web site: highest scores were among blogs (mean = 6.33; SD = 1.99) and medical, government, or university Web sites (mean = 4.82; SD = 2.27). However, the lowest scores went to commercial (mean = 2.37; SD = 2.60) or news and online media sites (mean = 3.52; SD = 2.31; analysis of variance P < .001). A mean design quality score of a possible 6 points was 3.62 (SD = 0.74).

FIGURE 3—

Content scores per Web site type: Accuracy of Weight Loss Information Search Engine Results on the Internet, 2012.

Note. med/uni/gov = medical, university, or government sites.

Reputable references were given in 46.6% of Web sites, with only 29.1% providing any hyperlinks, and 46.6% of sites included a date of last update. The mean reading level for the information was 8.66 years (SD = 2.31; minimum to maximum = 5.0–17.0; Table B, available as a supplement to the online version of this article at http://www.ajph.org). Unsubstantiated claims were made by more than half of the Web sites (54%), particularly regarding nutritional information. Medical or university Web sites were least likely to provide unsubstantiated claims (14%; Table C, available as a supplement to the online version of this article at http://www.ajph.org).

DISCUSSION

Our main finding was that online weight loss information likely to be accessed and most likely to be read by the public was suboptimal.

We observed that fewer than a fifth of Web sites included accurate information on more than 50% of the key information with regard to nutrition, physical activity, or behavioral strategies for weight loss. We also observed that no site was comprehensive in breadth or accuracy, because none scored close to the 16 point maximal score possible. Most sites that focused on pharmacotherapy and surgery tended to focus solely on these. We did not anticipate either the small number of medical sites (12 of 103) or the very small number of government or university sites (2 of 103) that appeared as a result of our queries. An informal search showed that they tended to be of high quality according to our scoring protocol. However, they mostly appeared in page 2 or 3 of Web searches and, thus, were far less likely to be viewed. This was the case for Web sites such as myplate.gov, which contained excellent information but did not appear near the top of searches. This was concerning because those who were more likely accessing the Internet for health information (e.g., minorities, people with higher body mass index, younger people) might be disproportionately exposed to incomplete and inaccurate information about weight loss.¹⁴

Although we had anticipated that medical sites and government or university sites would score well over these 3 criteria, we did not anticipate that blogs would score highly as well. In general, blogs provided accurate information pertaining to weight loss, in particular for nutritional advice, physical activity, and behavioral tips, but they also contained far more unsubstantiated claims than medical, government, or university sites. This might partially be explained by the fact that blogs were often written by authors interested in a given topic, who spent time collecting evidence from a variety of reliable and unreliable sources; the sites were also structurally better designed to contain the searched information on a single page. They also followed trends more closely and tended to be updated more frequently than government-sponsored Web sites. Because we did not lower the score if a page contained unsubstantiated claims, this did not negatively affect the scores of blogs. By contrast, sites such as WebMD tended to spread the content over multiple pages and, thus, scored less under our scoring protocol than a site that put all the information on a single page.

Most of the Web pages we evaluated had an average to good design, regardless of the type of site. Surprisingly, commercial sites and media or online news sites scored lower than blogs and government, university, or medical sites because of higher use of layering, and ads within and around the text. With regard to authorship, government, university, or medical sites scored significantly higher than blogs, commercial sites, and media sites. This was not very surprising because most pages from government, university, or medical sites were written by health care professionals, with appropriate credentials and proper authorship, whereas articles written for blogs, online media, and commercial sites often lacked authorship or credentials, or both. However, government, university, or medical sites did not top search engine result pages, and were thus less likely to be read than other Web sites that topped SERPs but provided far less accurate information pertaining to weight loss.

Strengths and Limitations

The main strength of our study was its focus on the information that is available online and that Internet users are most likely to explore when performing online queries. We did not focus on the information that is available but ranks low in SERPs, and is therefore very less likely to be seen. Another strength of our study was the development of a sensible and easy-to-use scoring rubric and protocol to efficiently evaluate the content of a Web site pertaining to weight loss. Although this rubric was weight loss specific, the overall structure could be preserved to quickly evaluate the quality of any health care Web site.

A potential weakness of our article is that we focused on only the first page resulting from a query. This, however, reflects actual use because the click patterns seen during Internet searches³² showed that when the first page did not contain information that was interesting to the user, the user was likely to try another site. This analysis approach also enabled investigators to have a rubric that could be used efficiently and effectively, regardless of the page and site structures, which reflects actual use by the public. Another potential weakness was that search results might vary over time. However, despite the inherent dynamic nature of the Internet, we could expect reasonable stability of search results because changes in the top 10 of a SERP occur gradually.⁴⁸ Over a period of 2 weeks, we could expect 20% changes in the top 10 results, but these changes included searches related to news events.⁴⁹ Another potential weakness was that we excluded forums from our analysis because they did not meet our inclusion criteria. Although the quality of advice for weight loss provided on such forums might be relatively high,⁵⁰ they experience the same problem as medical or government sites because they are less likely to be read because of the need for multiple clicks.

Conclusions

Our study showed that the majority of Web pages likely to be viewed by Internet users for weight loss were of suboptimal quality. Higher quality information provided by reputable organizations exists. Such organizations, particularly medical, government, or university organizations, need to improve their Web sites, with a particular emphasis on search engine optimization to ensure that their Web sites top searches pertaining to weight loss. These Web sites could also be significantly improved by providing more condensed information on a better overview of diet, weight loss, and physical activity. Health care providers might also want to make Web site recommendations to people interested in losing weight. In addition, providers could also take advantage of the simpler structure of blogs to effectively and efficiently disseminate accurate and concise weight loss information.