White Hat Bias: Examples of its Presence in Obesity Research and a Call for Renewed Commitment to Faithfulness in Research Reporting

Abstract

`White hat bias' (WHB) (bias leading to distortion of information in the service of what may be perceived to be righteous ends) is documented via quantitative data and anecdotal evidence from the research record regarding the postulated predisposing and protective effects respectively of nutritively-sweetened beverages and breastfeeding on obesity. Evidence of an apparent WHB is found in a degree sufficient to mislead readers. WHB bias may be conjectured to be fuelled by feelings of righteous zeal, indignation toward certain aspects of industry, or other factors. Readers should beware of WHB and our field should seek methods to minimize it.

INTRODUCTION

Scientific dialogue is dependent on fair and open presentation of data and evidence, yet recent years have raised concerns about bias in research practice. We present data and examples pertinent to a particular bias, a `white hat bias' (WHB), which we define to be bias leading to distortion of research-based information in the service of what may be perceived as righteous ends. We evaluate WHB in the context of two illustrative obesity topics, nutritively-sweetened beverage (NSB) consumption as a postulated risk factor [1] and breastfeeding as a postulated protective factor [8].

EXAMPLE 1 - Data on Citation Bias

If secondary reportings of original research misleadingly cite papers with statements that do inaccurately describe available evidence, then inaccurate beliefs may inappropriately influence clinical practice, public policy, or future research. Previously, [4], we observed that two papers [2,3] had both statistically and non-statistically significant results on body-weight, body mass index (BMI) or overweight/obesity status which allowed future writers to potentially choose which results to cite, and were also widely cited, permitting quantitative analysis of citations.

Cited vs Citing Papers

A Web of Science search (through October, 2008) yielded 195 and 45 papers citing James et al. [2] and Ebbeling et al. [3], respectively. We analyzed those in English (165 and 41, respectively).

James et al. [2] studied an intervention to decrease NSB consumption and adiposity among children. Dichotomized (overweight or obese vs neither overweight nor obese) and continuous (change in BMI) data were analyzed for statistical significance. The authors wrote:

“After 12 months there was no significant change in the difference in body mass index (mean difference 0.13, – 0.08 to 0.34) or z score (0.04, – 0.04 to 0.12). At 12 months the mean percentage of overweight and obese children increased in the control clusters by 7.5%, compared with a decrease in the intervention group of 0.2% (mean difference 7.7%, 2.2% to 13.1%).”

Ebbeling et al. [3] described an RCT of a 25 week NSB reduction program in adolescents and wrote:

“The net difference (in BMI), 0.14 ± 0.21 kg/m², was not significant overall.” They then report a subgroup finding: “Among the subjects in the upper baseline-BMI tertile, BMI change differed significantly between the intervention…and control…groups, a net effect of 0.75 ± 0.34 kg/m².”

Ebbeling et al. (p. 676) label the analysis in the total sample the “primary analysis.”

Data Coding and Analysis

Each paper citing either [2] or [3] was categorized (see Tables 1 & 2) based on how citing authors cited results related to body weight, BMI, or overweight/obesity outcomes from these two papers in their report. Papers citing James et al. were independently coded by this paper's authors (DBA or MBC). Any discrepancies were resolved by discussion. Papers citing Ebbeling et al. were scored by DBA and cross-checked by MBC. Proportions (with confidence intervals) were calculated (Tables 1 & 2). Exact binomial calculation tested the null hypothesis that the proportion citing papers in a misleading way that exaggerated the strength of evidence was equal to the proportion citing papers in a misleading way that diminished the strength of evidence; as such an equal proportion would suggest a lack of bias in the overall literature, even if not in any one paper.

Table 1.

Categorization of 165 papers citing James et al. (2004)

Score	A	B	C	D	E	F	G	H
No. of References in Category	14	74	2	21	2	1	1	50
Proportion (Exact CIs)6	0.127 (0.071 to 0.199)	0.644 (0.548 to 0.729)	0.017 (0.003 to 0.068)	0.183 (0.119 to 0.268)	0.017 (0.003 to 0.068)	0.009 (0.001 to 0.055)	0.009 (0.001 to 0.055)

Scoring Key:

A) Accurate - Described the non-significant result on continuous outcome (change in BMI) and described the significant result on the dichotomous outcome (overweight vs. non-overweight).

B) Mildly Misleading (positively) - Described the result of the intervention study as showing efficacy, benefit, or statistical significance for the dichotomous outcome of overweight status, without mentioning the non-significant result on the continuous outcome.

C) Moderately Misleading (positively) - Described the result of the intervention study as showing efficacy, benefit, or statistical significance on some weight related outcome without explicitly stating that it was on the proportion overweight per se.

D) Explicitly Misleading (positively) - Described, with a factually incorrect statement, that the result of the intervention for a continuous weight related outcome was significant or showed effectiveness.

E) Mildly Misleading (negatively) - Described the result of the intervention study as not showing efficacy, benefit, or statistical significance on the continuous measure of BMI, without mentioning the significant result on the dichotomous outcome.

F) Moderately Misleading (negatively) - Described the result of the intervention study as not showing efficacy, benefit, or statistical significance on some weight related outcome without explicitly stating that it was on the continuous measure of BMI.

G) Explicitly Misleading (negatively) - Described, with a factually incorrect statement, that the result for the dichotomous outcome was not significant or that a lack of effectiveness was shown for the dichotomous outcome.

H) Unscorable - Did not make explicit statements about the effects of the study, made statements that were too ambiguous to code, or made statements that were self-contradictory.

⁶Proportions and CIs are calculated with only categories A through G in the denominator.

Table 2.

Categorization of 41 papers citing Ebbeling et al. (2006)

Score	A	B	C	D	E	F	G
No. of References in Category	10	9	11	0	0	7	4
Proportion (Exact CIs)7	0.333 (0.173 to 0.528)	0.300 (0.147 to 0.494)	0.367 (0.199 to 0.561)	0.000 (0.000 to 0.116)	0.000 (0.000 to 0.116)

Scoring Key:

A) Accurate - Described both the non-significant result in the total sample and also the significant result in the heaviest subgroup.

B) Patently misleading over-positive - Described as positive on weight without mentioning anything about the result only being in heaviest children.

C) Mildly misleading over-positive - Described as positive among the heaviest children without explicitly mentioning that there was no significant result in the total sample.

D) Mildly misleading over-negative - Described the null result in the total sample without explicitly mentioning the significant result in the heaviest subgroup.

E) Patently misleading over-negative - Described as negative in a way that explicitly indicated that there were no significant effects even in sub-groups.

F) Not directly relevant - Did not make clear and explicit statements about the effects of the study.

G) Ambiguous as to whether category A or E applies.

⁷Proportions and CIs are calculated with only categories A through E in the denominator.

Citation Analysis Results

Results were quite consistent across papers citing either [2] or [3]. The majority, 84.3% for [2] and 66.7% for [3], described results in a misleadingly positive way to varying degrees (i.e., exaggerating the strength of the evidence that NSB reduction showed beneficial effects on obesity outcomes). Some were blatantly factually incorrect in their misleading statements, describing the result as showing an effect for a continuous obesity outcome whereas no statistically significant effect for continuous obesity outcomes was observed. In contrast, only four papers (3.5%) were negatively misleading (i.e., underplayed the strength of evidence) for [2] and none were negatively misleading for [3]. Only 12.7% and 33% of the papers accurately described complete overall findings related to obesity outcomes from [2] and [3], respectively.

To test whether the proportion of misleading reporting in the positive direction was equal to the proportion in the negative direction, we calculated the confidence interval on the proportion of misleading reportings in either direction that were positively misleading. This yields a proportion of 0.96 (95% CI: .903 to .985) for those citing [2] and 1.00 (95% CI: .832 to 1.000) for [3] and is significantly different from ½ for each (p < .0001) indicating a clear bias and potential for readers of the secondary literature to be deceived.

EXAMPLE 2 - Data on Publication Bias

NSB-Consumption

A meta-analysis on NSB consumption and obesity [6] found that estimated adverse associations were significantly smaller (i.e., less adverse) among industry-funded than non-industry-funded studies. One troubling conceivable explanation for this is that industry does something to bias results to make NSBs appear less harmful, but this is not the only conceivable explanation.

To examine this further, we requested, and Dr. Vartanian [6] graciously provided, his meta-analysis datafile. Focusing on cross-sectional studies because a large number had adiposity indicators as outcomes, we conducted publication bias (PB) detection analyses [12]. PB causes the sample of studies published to not constitute a representative sample of the relevant studies that hypothetically could have been published. With PB the probability of a study being published depends on its outcome. Typically, PB involves statistically significant studies having a higher likelihood of being published than non-statistically significant ones. Our analysis (Fig 1) shows a clear inverse association between study precision and association magnitude. This PB hallmark suggests that studies with statistically significant NSB findings are more likely to be published than are non-statistically significant ones. Interestingly, this bias appears to be present only for non-industry-funded research, suggesting that non-industry-funded scientists tend not to publish their non-significant associations in this area. Contrarily, industry-funded studies seem to all exceed some minimal level of precision. Thus, much of the reason for the smaller associations detected by Vartanian et al. [6] for industry funded research seems to be due to PB in non-industry-funded research. However, even after accounting for precision, the mean difference between the association magnitudes of the industry and non-industry funded studies is reduced by 33%, but not eliminated, suggesting that there may be competing biases operating in industry-funded research.

Figure 1.

Plot of sample effect sizes from cross-sectional studies of the association between sugar-sweetened beverage consumption and obesity indexes indicating publication bias among non-industry-funded studies.

Breast-Feeding

The World Health Organization [WHO; 7] published a meta-analysis on whether breastfeeding protects against obesity and also found evidence of PB. Figure 2 indicates this strikingly. We retrieved all of the papers from which data were obtained for Figure 2 to evaluate the impact of industry funding on this PB. None of the papers reported any industry funding or were obviously authored by authors employed by the infant formula industry. Thus, as with the NSB literature, there appears to be strong PB that is not apparently fueled by industry interests.

Figure 2.

Plot of the relation between association magnitude and study precision indicating publication bias in studies of breastfeeding and obesity (from [7]).

EXAMPLE 3 - Anecdotal Examples of Miscommunications in Press-Releases

Evidence suggests that “Press releases from academic medical centers often promote research that has uncertain relevance to human health and do not provide key facts or acknowledge important limitations” [5]. This is also occurring in the obesity field. For example, Ebbeling et al.'s paper [3] states, “change in body mass index (BMI) was the primary end point…The net difference, 0.14 ± 0.21 kg/m², was not significant overall,” and then reports the subgroup finding, “Among the subjects in the upper baseline-BMI tertile, BMI change differed significantly between the intervention…and control…groups.” Contrast this modest finding in a sample subset and the circumspect presentation in the original paper with the presentation in the press release issued by the authors' institution¹, which states “In randomized trial, a simple beverage-focused intervention led to weight loss” and never states that the primary analysis was not statistically significant.

When James et al. [2] was released, the press release issued on the BMJ website² stated “Discouraging children from drinking fizzy drinks can prevent excessive weight gain, according to new research available on bmj.com,” despite the facts that no analysis of weight change per se was reported and that the there was no significant effect on BMI change. Neither of these facts was mentioned in the press release.

Finally, in 2009, describing an observational epidemiologic study, UCLA issued a press release³ stating “…research released today provides the first scientific evidence of the potent role soda and other sugar-sweetened beverages play in fueling California's expanding girth.” One of the study authors was quoted in a subsequent news story stating “For the first time, we have strong scientific evidence that soda is one of the – if not the largest – contributors to the obesity epidemic."⁴ These statements are inaccurate and also unfair to all the authors of observational studies who published such research years before. The press release further stated “"The science is clear and conclusive” [emphasis added], despite the fact that this was correlational research and offered no statement to the reader to interpret the results as indicative of correlation and not necessarily causation.

EXAMPLE 4 – Inappropriate or Questionable Inclusion of Information

Research may also be misleadingly presented by inclusion of incorrect or questionable material in reviews. In our critical review of the WHO report on breastfeeding, we noted several examples (see [8], p. 597) where inspecting the original papers reviewed revealed that the WHO report authors selectively included some values from certain primary papers that led to stronger associations of breastfeeding with reduced obesity risk and excluded less impressive values from the same papers without explanation.

Similarly, Mattes et al. [4] noted that several reviews of NSB consumption and obesity inappropriately included a study [9] that was not actually a test of nutritive sweetener-containing solid food versus beverage nor of NSB consumption versus non-NSB consumption. Sweeteners were presented in both solid and beverage food forms. The original authors [9] wrote, “…subjects who were given supplemental drinks and foods [emphasis added] containing sucrose for 10 wk experienced increases in …body weight” and thus the study should never have been considered as evaluative of NSB effects. Mattes et al. [4] provide other examples of papers being inappropriately included in past reviews of NSB consumption and obesity.

CONCLUSION

Finding effective methods to reduce obesity is an important goal and appropriate evaluations of the strength of the evidence supporting procedures under consideration are vital. Sound evaluations critically depend on evidence being presented in non-misleading ways. Alarms have been sounded about dramatic rises in obesity levels, not without justification. And yet these alarms may also have roused passions. Certain postulated causes have come to be demonized (e.g., fast food, NSBs, formula feeding of infants) and certain postulated palliatives (e.g., consumption of fruits and vegetables, building of sidewalks and walking trails) seem to have been sanctified. Such demonization and sanctification may come at a cost. Such casting may ignite feelings of righteous zeal.

Some authors compare NSBs, fast-food, and other food and restaurant industry offerings to the tobacco industry [e.g., 10], suggesting for example comparisons between `Joe Camel' and `Ronald McDonald'⁵. To the extent that such comparisons inform us about important causes of obesity and how to reduce them, this is all to the good. But to the extent that such comparisons and other appeals to the passions inflame rather than inform, they may cloud judgment and decrease inhibitions against breaching ordinary rules of conduct. Historians indicate that during times of war, propagandists demonize (i.e., dehumanize) the enemy to inflame spirits and this facilitates some breaches of codes of conduct such as massacres [11]. Though inflaming the passions of scientists interested in public health is unlikely to provoke bloodshed, scientists have as a discipline our own code of conduct. Central to it is a commitment to faithful reporting, to acknowledging our studies' limitations, to evaluating bodies of evidence without selectively excluding information on the basis of its desirability – in short, a commitment to truthfulness. The demonization of some things and sanctification of others, though perhaps helpful in spurring social action, may be more harmful to us in the long run by giving unconscious permission to breach that code and thereby eroding the foundation of the scientific discipline.

Evidence presented herein suggests that at least one thing has been demonized (NSB consumption) and another sanctified (Breastfeeding), leading to bias in the presentation of research literature to other scientists and to the public at large, a bias sufficient to misguide readers. Interestingly, while many papers point out what appear to be biases resulting from industry funding, we have identified here, perhaps for the first time, clear evidence that white-hat biases can also exist in opposition to industry interests.

Whether WHB is intentional or unintentional, stems from a bias toward anti-industry results, significant findings, feelings of righteous indignation, results that may justify public health actions, or yet other factors is unclear. Future research should study approaches to minimize such distortions in the research record. We suggest that authors be more attentive to reporting primary results from prior studies rather than selectively including only part of the results, to avoiding PB, and to ensuring that their institutional press releases are commensurate with the studies described. Journal editors and peer-reviewers should also be vigilant and seek to minimize WHB. Clinicians, media, public health policy makers, and the public should also be cognizant of such biases and view the literature on NSBs, breastfeeding, and other obesity-related topics more critically.