Quality of weight loss advice on Internet forums

Abstract

Background:

Adults use the Internet for weight loss information, sometimes by participating in discussion forums. Our purpose was to analyze the quality of advice exchanged on these forums.

Methods:

This was a retrospective analysis of messages posted to 18 Internet weight loss forums during one month in 2006. Advice was evaluated for congruence with clinical guidelines; potential for causing harm; and subsequent correction when it was contradictory to guidelines (erroneous) or potentially harmful. Message- and forum-specific characteristics were evaluated as predictors of advice quality and self-correction.

Results:

Of 3368 initial messages, 266 (7.9%) were requests for advice. Of 654 provisions of advice, 56 (8.6%) were erroneous and 19 of these 56 (34%) were subsequently corrected. Forty-three (6.6%) provisions of advice were harmful and 12 of these 43 (28%) were subsequently corrected. Messages from low-activity forums (fewer messages) were more likely than those from high-activity forums to be erroneous (10.6% vs. 2.4%, p<0.001) or harmful (8.4% vs. 1.2%, p<0.001). In high-activity forums, 2 of 4 (50%) erroneous provisions of advice and 2 of 2 (100%) potentially harmful provisions of advice were corrected by subsequent postings. Compared to general weight loss advice, medication-related advice was more likely to be erroneous (p=0.02) or harmful (p=0.01).

Conclusions:

Most advice posted on highly-active Internet weight loss forums is not erroneous or harmful. However, clinical and research strategies are needed to address the quality of medication-related advice.

INTRODUCTION

Almost two thirds of US adults are obese or overweight.¹ Weight loss can attenuate many adverse consequences of excess body weight, such as diabetes, hypertension, and cardiovascular disease.² Clinical guidelines recommend dietary restriction and physical activity for weight loss, with pharmacologic or surgical therapy added when obese individuals fail to lose weight through lifestyle changes.^2,3 However, only one fifth of people trying to lose weight restrict their caloric intake and exercise at least 150 minutes per week,⁴ while millions use nonprescription weight loss medications.⁵ These data suggest that the choice of weight loss modality for many Americans is not consistent with weight loss guidelines.

Although physician advice affects the patient's choice of weight loss modality,⁴ people also make health choices based on information they find on the Internet.⁶ Since U.S. adults are five times more likely to first seek health information from the Internet rather than from health care providers,⁷ and weight loss is one of the most popular search topics,^8-10 the potential impact of the Internet in this area is enormous.

Internet forums allow participants to carry on asynchronous conversations by posting messages and replies.¹¹ We previously found that information posted on a breast cancer Internet forum was generally accurate, with most inaccurate statements rapidly corrected by subsequent posts.¹² However, it is unknown if this phenomenon holds true for weight loss forums.

We sought to explore whether clinicians should recommend Internet weight loss forums to patients trying to lose weight. Therefore, we studied how adults request and provide weight loss advice on Internet forums. We evaluated the advice for congruence with weight loss guidelines and potential for causing harm if the advice were followed. We also determined whether bad advice was corrected in subsequent postings.

METHODS

We identified forums by searching with the term “weight loss forum” on March 22, 2006 using Google (http://www.google.com), the most popular search engine in the US.¹³ This approach identified forums likely to be found by patients, rather than a random sample of forums. From the first 50 search results, we selected freely-accessible English-language websites containing weight loss forums pertaining to general weight loss or weight loss modalities of diet, exercise, medications, or surgery. The forums had to have at least ten initial messages in a specific month in early 2006, the time period to which we restricted our study (exact month withheld to preserve confidentiality). An initial message is a message which starts a discussion thread.

The units of analysis were requests for and provisions of weight loss advice. Requests for advice were included if they could be addressed by weight loss guidelines from the National Heart, Lung, and Blood Institute² or the American College of Physicians³ (Appendix 1). We included questions on how to start losing weight (“Need help getting started!”), choose weight loss modalities (“Should I take a diet pill?), and perform weight loss modalities (“How much should I exercise?”). Requests for advice were excluded if they fell outside the realm of the guidelines (“Do oranges have more calories than apples?”) or if participants stated they were younger than 18 years old.

To assess inter-observer reliability of dichotomous outcomes, we computed Prevalence and Bias Adjusted Kappa (PABAK) which adjusts for rater bias, expressed as the Bias Index (BI), and relative probabilities of “yes” and “no” responses, expressed as the Prevalence Index (PI) . We used PABAK to adjust for the low prevalence of some responses. Like kappa, PABAK=0 represents chance agreement, while PABAK=1 implies maximum agreement. BI and PI values range from 0 to 1 (absolute value) with increasing BI and PI values respectively indicating increasing rater bias or prevalence.^14,15

K.H. selected requests for advice which could be addressed by the guidelines. To evaluate the reliability of this process, a random sample of 12% of all initial messages, stratified by forum, was independently reviewed by K.F. Agreement was 85.4%, kappa 0.41, and PABAK 0.71 (PI = − 0.72 and BI = 0.11), indicating very good agreement after adjusting for prevalence and bias. K.H. determined whether advice was in response to questions about diet, exercise, medications, surgery, or general weight loss (nonspecific question or multiple weight loss modalities).

Provisions of advice in reply messages were evaluated for congruence with weight loss guidelines, and categorized as supported by guidelines (“accurate”), contradictory of guidelines (“erroneous”), or not addressed by guidelines. If any part of the advice was erroneous, the advice was erroneous. If none of the parts were addressed by guidelines, the advice was considered not addressed. If all parts were accurate, or if some parts were accurate while other parts were not addressed, the advice was accurate. Advice was further categorized as potentially harmful (“harmful”) or not likely harmful (“not harmful”). In the absence of a gold standard for evaluating the potential harmfulness of advice or information, we used the clinical judgment of two independent reviewers. Advice was considered potentially harmful if the requestor of advice would likely come to harm by following that advice. The reviewers determined whether erroneous or harmful advice was corrected in a subsequent post within the same thread.

K.H. and K.F. independently assessed the advice in two separate stages to minimize the influence of correction of advice when evaluating whether the advice was concordant with the guidelines. In the first stage, advice was divided into the three categories (“accurate,” “not addressed,” or “erroneous”), with agreement 83% and kappa 0.58, indicating moderate agreement. When advice was categorized as erroneous or not erroneous, agreement was 92%, kappa 0.47, and PABAK 0.83 (PI = 0.83 and BI = 0.003), indicating very good agreement after adjusting for prevalence and bias. Advice was categorized as harmful or not harmful during this first stage. Percent agreement was 94%, kappa 0.51, and PABAK 0.89 (PI = 0.88 and BI = − 0.002). In the second stage, the reviewers determined whether erroneous or harmful advice was corrected in a subsequent post. For correction of erroneous advice, agreement was 80%, kappa 0.57, and PABAK 0.61 (PI = − 0.30 and BI = − 0.02), indicating good agreement. For correction of harmful advice, agreement was 88%, kappa 0.72, and PABAK 0.77 (PI = − 0.42 and BI = − 0.02). Disagreements were resolved by consensus.

We performed chi square (χ²) tests of independence to examine the relationship between the topic of advice (diet, exercise, medications, surgery, or general) and occurrence of erroneous advice or harmful advice. The reference category was general advice.

We examined relationships between the dependent variables of erroneous advice, harmful advice, correction of erroneous advice, and correction of harmful advice and the forum-specific independent variables of forum activity, presence of moderators as participants, and age of forum. Independent variables were dichotomized. Two forums containing 56.4% (16731 of 29684) of all messages were high-activity forums. The remaining were low-activity forums. Since these two high-activity forums were from the SparkPeople website, analysis of relationships between activity and dependent variables may have been confounded by website-specific characteristics. Therefore, we employed the same sequence of logistic regression analyses using an alternative cutoff point between high- and low-activity forums. This alternative cutoff point was 1000 total messages per month, since it lay within a large gap dividing lower and higher activity forums. Six forums from five websites were categorized as high-activity with this cutoff, accounting for 87.9%% (26104 of 29684) of all messages.

For moderators as participants, forums were categorized as with or without moderators who can post messages. For age, forums were categorized as starting before February 2004 or during February 2004 and later, with data available for 16 of 18 forums.

We performed standard logistic regression analysis for each dependent variable, followed by hierarchical regression analyses if the omnibus χ² from the standard logistic regression analysis was significant.¹⁶ When logistic regressions with all three predictors did not converge, we used relevant combinations of two predictors. We entered in the final logistic regression model the independent variables which were consistently significantly related to the dependent variable. Logistic regressions for erroneous and harmful advice were performed with the full data set. Logistic regressions for correction of erroneous and harmful advice were limited to forums with erroneous or harmful advice. Statistical analyses were performed with the SPSS 14.0 (SPSS Inc., Chicago, IL). Level of significance was set at α < 0.05.

The protocol was deemed exempt by the Institutional Review Board at the University of Texas Health Science Center at Houston.

RESULTS

Eighteen weight loss forums on 13 websites contained 29684 messages during the month, with an average of 1649 messages per forum (Appendix 2). Of 3368 initial messages, 3102 (92.1%) requested motivational support, requested advice outside the realm of the weight loss guidelines, or were not requests. The other 266 (7.9%) were requests for advice that could be addressed by the guidelines, 245 (92.1%) of which were answered by at least one provision of advice. There were 654 responses containing advice, with an average of 2.5 provisions of advice for each request.

With respect to the weight loss guidelines, 504 of 654 (77.1%) of provisions of advice were accurate, 94 (14.4%) were not addressed by guidelines, and 56 (8.6%) were erroneous. Only 43 (6.6%) provisions of advice were considered potentially harmful. Erroneous advice was more likely than other advice to be harmful (χ²=398.91, df=2, p<0.001) (Table 1). The twentyfour (3.7%) provisions of advice posted by moderators did not differ from other advice in accuracy or potential harmfulness.

Table 1.

Quality of advice in messages (n=654) on Internet weight loss forums

Quality of advice*	N (%)
Accurate	504 (77.1)
Harmful	0 (0)
Not harmful	504 (100)
Not addressed by guidelines	94 (14.4)
Harmful	4 (4)
Not harmful	90 (96)
Erroneous	56 (8.6)
Harmful	39 (70)
Not harmful	17 (30)
Total	654 (100)

^*χ²=398.91, df=2, p<0.001 for relationship between potential harm and congruence with guidelines

Examples of erroneous or harmful advice were encouragement to take over-the-counter weight loss medication which could have harmful effects, or to take it despite inadequate attempts at lifestyle changes; statement that only the types of foods consumed (but not total caloric intake) is relevant to weight loss attempts; and encouragement to try diet only, without exercise, for weight loss.

Of the 56 provisions of erroneous advice, 19 (34%) were subsequently corrected within the same thread. One correction was posted by a moderator. The average time to correction was 17 hours and 31 minutes. The other 37 (66%) provisions of erroneous advice were not corrected, with four of these posted during the last two days of the month. Twelve of 43 (28%) provisions of harmful advice were corrected, with an average time to correction of 18 hours and 3 minutes. The other 31 (72%) provisions of harmful advice were not corrected, three of which were posted during the last two days of the month. No corrections of harmful advice were posted by moderators.

Message characteristics and quality of advice

Of 654 provisions of advice, 363 (55.5%) addressed requests for general weight loss advice; 113 (17.3%) addressed requests for dietary advice; 70 (10.7%) addressed requests for exercise advice; 101 (15.4%) addressed requests for medication advice; and seven (1.1%) addressed requests for bariatric surgery advice. The relationship between topic and congruence with guidelines was statistically significant (χ²=24.25, df=2, p=0.002) (Table 2). Compared to advice related to general weight loss strategies, medication-related advice was more likely to be erroneous or harmful (Table 3).

Table 2.

Congruence with guidelines for advice in messages (n=654) on Internet weight loss forums according to topic of advice.

	Topic of Advice
	General	Diet	Exercise	Medication	Surgery
	(n=363)	(n=113)	(n=70)	(n=101)	(n=7)
	N (%)	N (%)	N (%)	N (%)	N (%)
Congruence with guidelines *
Accurate	275 (75.8)	96 (85)	58 (83)	68 (67)	7 (100)
Not Addressed	54 (14.9)	14 (12)	11 (16)	15 (15)	0 (0)
Erroneous	34 (9.4)	3 (3)	1 (1)	18 (18)	0 (0)

^*χ²=24.25, df=2, p=0.002 for relationship between topics of advice and congruence with guidelines

Table 3.

Odds ratios of erroneous and harmful advice according to topic of advice in messages (n=654) on Internet weight loss forums.

	Erroneous advice			Harmful advice
Topic of advice	OR	95% CI	p		OR	95% CI	p
General	1.00	--	--		1.00	--	--
Diet	0.26	0.08-0.88	0.03		0.37	0.11-1.25	0.11
Exercise	0.14	0.02-1.04	0.06		0.00
Medication	2.10	1.13-3.90	0.02		2.36	1.19-4.67	0.01
Surgery	0.00				0.00

Forum characteristics and quality of advice

As shown in Tables 4A and 4B, messages from low-activity forums were more likely than those from high-activity forums to have erroneous advice (10.6% vs. 2.4%; logistic regression: χ²=13.33, p<0.001, OR=4.79, 95% CI 1.71–13.46) or harmful advice (8.4% vs 1.2%; logistic regression: χ²=13.84, p<0.001, OR=7.46, 95% CI 1.78–31.19). There were significant correlations (p<0.001) between activity and moderators (φ=0.191); activity and forum age (φ=−0.485); and moderators and forum age (φ=0.215).

There were too few cases of correction to conduct valid statistical analyses of the effects of forum characteristics on correction of erroneous or harmful advice. However, 2 of 4 (50%) provisions of erroneous advice and both provisions of harmful advice were corrected in the high-activity forums.

When the cutoff point between high- and low-activity was set at 1000 total messages per month, results were similar. Messages from low-activity forums were still significantly more likely to have erroneous or harmful advice than messages from high-activity forums.

DISCUSSION

When participants of Internet weight loss forums ask for advice which can be addressed by clinical guidelines, they generally receive accurate and non-harmful advice. Although erroneous or harmful advice was uncommon, it was most likely to occur in response to medication-related questions and on low-activity forums. Two-thirds of erroneous or harmful advice posted on high-activity forums was corrected by subsequent postings. However, the quality of medication-related advice underscores the need for clinical and research strategies to explore the role of these forums as a weight loss intervention.

This is the first study to systematically examine the quality of advice on Internet weight loss forums. We analyzed multiple forums likely to be encountered by consumers searching for weight loss information online. Previous studies of health topics analyzed only a single forum.^12,17,18

Our study has several limitations. First, we analyzed only one month of messages. This allowed us to include enough forums to gather forum-specific data but our data may include temporal bias. Second, we used one search term to identify forums. Third, although we used established weight loss guidelines to evaluate accuracy of advice, other guidelines may have different recommendations. Fourth, given the anonymous and informal nature of these forums, we could not evaluate the qualifications of those who gave advice. However, these forums are designed for peer-to-peer interaction rather than professional consultation; moderators rarely posted or corrected advice. Fifth, it was sometimes difficult to evaluate advice without knowing more about the person requesting advice. However, we analyzed the same information available to participants who give advice. Lastly, evaluating the quality of advice is necessarily subjective, despite the use of weight loss guidelines as a standard for accuracy. Therefore, we used two independent reviewers to enhance the reliability and reduce the subjectivity of the evaluation.

Comparing studies of information on Internet forums is difficult due to differences in design. In a breast cancer forum, 0.22% of postings (not necessarily advice) were false or misleading and 70% (7 of 10) of these were corrected.¹² In our weight loss forums, 34% of erroneous advice and 28% of harmful advice were corrected, but these proportions were higher in high-activity forums.

Advice related to weight loss medication was more likely than other types of advice to be erroneous or harmful. Medications have a limited role in weight loss² but are popular,^5,19 suggesting that some of their use is inappropriate. In addition, advice from high activity forums was less likely than advice from low activity forums to be erroneous or harmful. We hypothesize that having many contributors protects against errors, just as in the online encyclopedia Wikipedia (http://www.wikipedia.org) and the “open source” Linux operating system. High-activity forums have the additional benefit of providing rapid responses to requests for advice. In contrast, the relatively infrequent advice from forum moderators was no better than advice from non-moderators. This suggests that advice quality depends more on the frequency of messages in the forum rather than characteristics of individual participants.

Should clinicians encourage patients who wish to lose weight to participate in Internet forums? Based on our findings, we recommend that clinicians: (1) refer patients to high-activity forums, such as the “Diet and Nutrition” or “Fitness and Exercise” forums on http://www.SparkPeople.com or other forums with more than 1000 messages per month; (2) refer patients to forums after discussing the proper role of weight-loss medications or warn them not to follow medication-related advice on the forums. Alternatively, if no erroneous advice can be tolerated, avoid referring patients to forums. In our opinion, it is not realistic to expect 100% accuracy from any information source (including clinicians). Our recommendations are intended to harness the positive aspects of the forums while providing some protection against potentially bad medication-related advice.

Our findings reveal opportunities for future research. The effect of medication-related advice on behavior, weight loss, and adverse events should be examined. Secondly, in addition to advice, Internet forums can also provide social support,²⁰ and social support is helpful for weight loss.^21,22 Therefore, the quality of social support exchanged among Internet weight loss forum participants warrants further study. Most importantly, outcome studies are needed to assess the impact of forums as lone or adjunctive interventions.²³ Referral to Internet weight loss forums could be one arm of a randomized trial comparing weight loss interventions.

In conclusion, highly-active Internet weight loss forums contain a low overall prevalence of erroneous or harmful advice. However, clinicians should consider approaches to minimize the effect of medication-related advice. Further investigation is needed to clarify the role of these forums in the management of overweight and obesity.

Appendix 1

Selected recommendations from weight loss guidelines for evaluating advice on forums

1998 National Heart, Lung, and Blood Institute obesity and overweight management guidelines2 Initial goal should be to reduce body weight by approximately 10 percent from baseline, within 6 months Rate of weight loss should be 0.5 to 2 pounds per week Aim for a caloric deficit of 500 to 1,000 kilocalories (kcal) per day Total caloric intake should be approximately 1,000 to 1,200 kcal per day for women and 1,200 to 1,500 kcal per day for men Reduction in dietary fat and carbohydrates will usually be needed to achieve a desirable caloric deficit Reduction in dietary fat alone is not sufficient to produce weight loss unless total caloric intake is reduced Increased physical activity is a key component of weight loss efforts All adults should aim to accumulate at least 30 minutes of moderate-intensity physical activity on most, and preferably all, days of the week Pharmacotherapy should not be considered until a combination of low calorie diet and increased physical activity has been maintained for at least 6 months Pharmacotherapy can be added to dietary therapy and physical activity for some patients with body mass index (BMI) of ≥ 30 kg/m2 with no concomitant risk factors or diseases, and for patients with BMI of ≥ 27 kg/m2 with hypertension, dyslipidemia, coronary heart disease, type 2 diabetes, or obstructive sleep apnea

2005 American College of Physicians obesity management guidelines3 Medications approved for weight loss by the U.S. Food and Drug Administration include: sibutramine, phentermine, diethylpropion, orlistat. Other options for medical treatment of obesity include fluoxetine and bupropion. Weight loss surgery can be considered for patients with BMI of ≥ 40 kg/m2 who have not lost weight despite diet and physical activity, with or without pharmacotherapy, and who have developed hypertension, impaired glucose tolerance, diabetes mellitus, hyperlipidemia, or obstructive sleep apnea After weight loss surgery, patients should resume their previous eating habits and they should continue exercising

Appendix 2

Number of messages, requests for advice, and provisions of advice on weight loss forums.

Website	Forum name	All messages	Initial messages	Requests for advice	Provisions of advice
3fatchicks	Weight loss surgery	137	16	2	8
	Support	2720	214	20	65
About	Ask & Answer	129	13	2	3
	General Discussion	1760	86	5	17
Calorie-Count	None	2758	289	48	133
CalorieKing	General Chat	2135	143	5	21
Calories Per Hour	General Discussions on Diet and Weight Loss	416	54	12	37
	Please Help Me, I'm Stuck!	127	25	10	16
Featherish	Weight loss challenges	243	38	11	21
	Diets and healthy eating plans	349	66	9	19
HealthBoards	Weight Loss	942	175	45	96
HealthyWeightForum	General	223	39	2	3
MedHelp	Weight Loss	218	54	12	23
SparkPeople	Diet and Nutrition	9423	1120	36	99
	Fitness and Exercise	7308	952	33	66
Tbfinc	Diet Help	33	13	4	4
Weight-loss. Fitness	The club	603	42	8	21
Weight-loss- forums	Lipovarin	160	29	2	2
	TOTAL	29684	3368	266	654