Source of bias in sugar-sweetened beverage research: a systematic review

Abstract

Objective

Financial conflicts of interest involving the food industry have been reported to bias nutrition studies. However, some have hypothesized that independently funded studies may be biased if the authors have strong a priori beliefs about the healthfulness of a food product (‘white hat bias’). The extent to which each source of bias may affect the scientific literature has not been examined. We aimed to explore this question with research involving sugar-sweetened beverages (SSB) as a test case, focusing on a period during which scientific consensus about the adverse health effects of SSB emerged from uncertainty.

Design

PubMed search of worldwide literature was used to identify articles related to SSB and health risks published between 2001 and 2013. Financial relationships and article conclusions were classified by independent groups of co-investigators. Associations were explored by Fischer’s exact tests and regression analyses, controlling for covariates.

Results

A total of 133 articles published in English met inclusion criteria. The proportion of industry-related scientific studies decreased significantly with time, from approximately 30 % at the beginning of the study period to <5 % towards the end (P=0·003). A ‘strong’ or ‘qualified’ scientific conclusion was reached in 82 % of independent v. 7 % of industry-related SSB studies (P<0·001). Industry-related studies were overwhelmingly more likely to reach ‘weak/null’ conclusions compared with independent studies regarding the adverse effects of SSB consumption on health (OR=57·30, 95 % CI 7·12, 461·56).

Conclusion

Industry-related research during a critical period appears biased to underestimate the adverse health effects of SSB, potentially delaying corrective public health action.

The nature of financial sponsorship may bias nutrition research results, hindering the pursuit of scientific truth and harming public health. Numerous studies report strong associations between funding of research by the food industry and scientific outcomes favourable to sponsors’ interests^{( 1 – 4 )}. However, these observational findings cannot prove that industry sponsorship causes bias. Some investigators have proposed an alternative explanation for the observed relationship between sponsorship and outcome, wherein investigators with independent funding may be biased against food products perceived to be unhealthful (the so-called ‘white hat bias’)^{( 5 )}. As the toll of diet-related disease continues to rise and public health experts increasingly advocate for national food policy changes, it is crucial that any threats to scientific integrity be investigated and, where possible, remediated.

The case of sugar-sweetened beverages (SSB) provides a special opportunity to compare the influence of industry-related v. independent funding, as the field transitioned from scientific uncertainty to consensus over a relatively short period. During the 1990s, dietary fat was considered the primary cause of weight gain and obesity-associated diseases, whereas added sugars were of only secondary concern to ‘special populations’ such as children at risk of dental caries^{( 6 )}. Indeed, added sugars were commonly believed to be benign or even protective against obesity, because they displaced energy from fat in the diet^{( 7 , 8 )}.

Although concerns about SSB consumption were raised publicly as early as the 1990s (e.g. in Liquid Candy, a report by the Center for Science in the Public Interest initially published in 1998)^{( 9 )}, several prospective observational studies and clinical trials first linked dietary sugar in the form of SSB to obesity or diabetes in the early 2000s^{( 10 – 12 )}, stimulating interest in public health initiatives to reduce SSB consumption through educational campaigns, taxes and other measures. In 2003, the Joint WHO/FAO Expert Consultation on Diet, Nutrition and the Prevention of Chronic Diseases recommended that free sugars should account for no more than 10 % of energy intake to prevent chronic disease, with elimination of SSB from the diet as one mechanism by which to achieve that target^{( 13 )}. In 2004, the American Academy of Pediatrics Committee on School Health warned health-care professionals, parents and schools of the adverse effects of SSB consumption in children^{( 14 )}. In 2009, the American Heart Association linked increased added sugar in the American diet, predominantly from SSB, to increased cardiometabolic disease and proposed a limit on added sugars of 628 kJ/d (150 kcal/d) for men and 418 kJ/d (100 kcal/d) for woman (approximately 5 % of total energy)^{( 15 )}. The US Department of Agriculture 2015 Dietary Guidelines Advisory Committee determined that the strength of the evidence linking added sugars and especially SSB to obesity was strong, and that SSB increased risk for diabetes at least partially independently of body weight (also with a strong level of evidence)^{( 16 )}. Indeed, the widespread interest in taxing SSB, rather than all sources of ‘empty calories’, reflects a common understanding of the uniquely adverse health impacts of these products^{( 17 )}. Thus, by the early to mid-2010s, a consensus had emerged among major nutrition-related professional and governmental organizations regarding SSB.

The first decade of the 21st century also saw extensive sponsorships by the sugar and SSB industry of individual scientific researchers, research organizations and professional organizations. The results of ensuing sponsored research were used by industry-related groups to lobby against policy proposals to limit SSB consumption^{( 18 , 19 )}. For example, PepsiCo Inc., Coca-Cola Co. and the American Beverage Association spent more than $US 40 million on lobbying in 2009, as the US Congress considered levying an SSB tax^{( 18 )}.

The present study aimed to examine the ‘natural history’ of SSB research funding, and its potential impact on public health policy, as the field emerged from uncertainty to consensus. Specifically, we hypothesized that industry-related articles: (i) were relatively prevalent soon after publication of the initial findings linking SSB to adverse health effects, and then decreased in prevalence as scientific consensus emerged; (ii) were disproportionately cross-sectional observational studies (a weak design) and systematic reviews or meta-analyses (a design especially susceptible to multiple biases)^{( 20 )}; and (iii) resulted in weaker or less negative scientific conclusions regarding the adverse effects of SSB consumption on health.

Methods

Overview

Two investigators independently selected for analysis articles published between 2001 and 2013, a period of time during which the scientific community transitioned from uncertainty to general consensus regarding the adverse health effects of SSB. Funding source and author conflicts of interest were classified by an investigator who had no knowledge of the strength of the study’s conclusion. The strength of the scientific conclusions for each manuscript, with respect to potential adverse effects of SSB on health, was rated by two investigators who were masked to article authorship and industry relationship. Associations between industry relationship and the strength of the article conclusion were calculated.

Manuscript selection

To obtain a broad view of the literature, we included a range of articles in four categories involving the health effects of SSB: (i) systematic reviews and meta-analyses; (ii) cross-sectional observational studies; (iii) prospective observational studies; and (iv) interventional studies. Commentaries, editorials and letters were excluded.

Articles were identified using a PubMed search. We used the following terms to identify the beverages of interest: ‘sugar sweetened’, ‘nutritively sweetened’, ‘nutritive’, ‘fruit’, ‘carbonated’ or ‘soft’ AND ‘beverage’, ‘drink’, ‘soda’ or ‘cola’. We used the following terms to identify the health risks of interest: ‘obesity’, ‘weight’, ‘BMI’, ‘cardiovascular’, ‘cholesterol’, ‘metabolic’, ‘blood pressure’, ‘inflammation’ or ‘diabetes’. Studies not focused primarily on SSB, not published in English, or without objective and specific anthropometric data or biometric end points as primary outcome measures were excluded from the analysis. Thus, we did not include articles exclusively examining general dietary patterns, even if SSB consumption was a component of that pattern. Two authors (E.A.L. and D.S.L.) independently assessed study eligibility and any disagreement was resolved by a third author who was blinded to study authorship, declared conflicts of interest and funding source.

Classification of funding source and declared conflicts of interest

One investigator (E.A.L.) independently abstracted information regarding financial sponsorship for each selected manuscript. Another investigator (D.S.L.), masked to the article context (including conclusion strength ratings), used the abstracted information to categorize funding source as: (i) ‘industry-related’, including for-profit and non-profit affiliations with the SSB industry (e.g. American Beverage Association); (ii) ‘independent’, for government, university, independent foundations, philanthropies and other philanthropic organizations without direct association with the SSB industry; or (iii) ‘both.’

As an additional method to identify conflicts of interest prior to consistently enforced disclosure requirements^{( 21 )}, we used the Integrity in Science Database^{( 22 )} to assign: (i) ‘conflict present’, if the first or last author ever received funding from an SSB industry-sponsored organization or had been affiliated with such an organization; or (ii) ‘conflict absent’.

Conclusion strength ratings

Two investigators (C.B.E. and S.L.G.) independently rated each article and determined the strength of the conclusion according to study design, and met to resolve discrepancies, using the following scoring guidelines. The articles were redacted for authorship, funding sources, acknowledgements and statements of conflict of interest.

• 1.Systematic reviews and meta-analyses were scored as ‘strong’ for a definitive, strong or indicative conclusion regarding an adverse effect; ‘qualified’ for a conclusion overall supportive of, or suggestive of an effect, or an effect in a subgroup; or ‘weak/null’ for an inconclusive, equivocal, weak or null overall conclusion, or only a speculative conclusion in a subgroup.
• 2.Observational studies were scored as ‘strong’ for a strong independent association with adverse health outcomes; ‘qualified’ for an association in a subgroup; or ‘weak/null’ for no or an unexpectedly weak association.
• 3.Interventional studies were scored as ‘strong’ for evidence of a strong adverse effect of SSB; ‘qualified’ for an effect in a subgroup or secondary analysis; or ‘weak/null’ for no effect or an unexpectedly weak effect.

Thus, regardless of study design, ‘strong’ and ‘qualified’ ratings reflected the raters’ evaluation of the different degrees of positive results, whereas ‘weak/null’ indicated unexpectedly weak associations or effects.

Statistical treatment of data

We pooled all article types and evaluated the association between the strength of article conclusion and industry relationship using a two-sided Fisher’s exact test. We also calculated logistic regressions and adjusted for article type and year of publication. We controlled for article type by creating three indicator variables and specifying cross-sectional studies as the reference category. For the association between study design and industry relationship, we excluded meta-analyses and systematic reviews, which referenced articles captured in the three other categories. To evaluate change in the proportion of industry-related studies over time, we grouped data by year and calculated a linear regression on data from 2003 to 2013. (We excluded the years 2001 and 2002 from that analysis, as there were too few articles published during that time (only one) to provide meaningful data for the regression.) We collapsed studies directly sponsored by industry and studies conducted by researchers who had received past financial support from industry into a single category.

Results

A total of 1851 potentially eligible articles were identified, from which 133 were selected for study inclusion (Fig. 1). Of these, sixteen were systematic reviews, fifty-seven were cross-sectional observational studies, forty-six were prospective observational studies and fourteen were interventional studies. Regarding evidence category, thirty-five were rated as ‘weak/null’, thirty-four as ‘qualified’ and sixty-four as ‘strong’.

Fig. 1.

Flowchart of articles included in the present review

The proportion of industry-funded studies compared with independently funded studies within each study design varied significantly (P=0·014). The meta-analyses and systematic reviews category contained the highest proportion of industry-funded studies (31 %), followed by cross-sectional observational studies (14 %), interventional studies (7 %) and prospective observational studies (0 %). As illustrated in Fig. 2, the proportion of industry-related scientific studies decreased significantly with time, from approximately 30 % at the beginning of the study period to <5 % towards the end (P=0·003).

Fig. 2.

Decreasing proportion of industry-related scientific studies on sugar-sweetened beverages () over the study period (, linear regression line: P<0·001). (Too few articles were published in 2001 and 2002 to provide meaningful data for the regression)

Supplemental Fig. 1 (see online supplementary material) depicts industry relationship and conclusion strength by individual study among the four article categories. A ‘strong’ or ‘qualified’ scientific conclusion was reached in 82 % of independent v. 7 % of industry-related SSB studies (P<0·001). Industry-related studies were overwhelmingly more likely to reach ‘weak/null’ conclusions compared with independent studies (OR=57·30, 95 % CI 7·12, 461·56), a difference that remained significant after adjustment for article type and publication year (OR=58·82, 95 % CI 6·32, 500·00).

Discussion

The present study investigated the association between industry relationship and strength of conclusions among 133 articles on the health effects of SSB published between 2001 and 2013, a period during which scientific consensus emerged from uncertainty. We found that industry-related articles reached much weaker evidence conclusions compared with independent studies. In the light of current consensus, these results imply that industry-related studies as a group systematically misperceived and underestimated the true health effects of SSB. Thus, among these two potentially opposing influences, our results suggest that industry-related bias predominated during a critical time in this area of scientific inquiry.

In addition, all but one industry-related article had designs that were either inherently weak (cross-sectional observational) or highly susceptible to bias (systematic review/meta-analysis)^{( 20 )}. According to several investigations, meta-analysis findings are less likely to be true when available data are limited, populations and data collection methodology are variable, and financial or other competing interests are involved – all factors that pertain to SSB research during the study period^{( 23 , 24 )}.

This potentially biased body of work, published predominantly during early stages of investigation, was used by the SSB industry to undermine policy measures to limit consumption – providing further evidence of a strategy by which producers of unhealthful products subvert science to maintain profitability^{( 25 , 26 )}. We cannot make a quantitative assessment as to actual public health harms from industry lobbying backed by industry-related research. However, even a 3-year delay in SSB taxes in several large states could have translated into the consumption of many billions of additional servings, with important impacts on rates of obesity, diabetes and CVD^{( 17 )}.

We also found that industry-related research declined in relative and absolute amounts during the last half of the study period, perhaps reflecting a recognition by industry that relatively weak research would not provide a ‘return on investment’ as scientific consensus on the adverse effects of consuming SSB took shape. Thus, the most relevant impacts of industry-related research may relate to the pace of scientific discovery, more so than the ultimate nature of that discovery.

Our study has several important limitations. First, we considered the consensus among major nutrition-related professional associations and the 2015 US Department of Agriculture Dietary Guidelines Advisory Committee as gold standard evidence for the adverse health effects of SSB. However, the precise size of these effects, and subgroup sensitivity, remain areas of ongoing investigation. Moreover, the level of evidence necessary for scientific proof (v. public health action)^{( 27 )} represents a legitimate subject of debate. Second, we did not objectively measure scientific quality and consequently cannot infer that any individual industry-related study is biased. Neither can we exclude the presence of white hat bias among individual independent studies in our sample. Indeed, both sources of bias may be present. (Measures to minimize any bias in the present study include use of pre-established, objective criteria for article selection and use of separate teams to assess study outcomes and funding, with appropriate blinding.) In addition, we cannot assess the relative effects of industry v. white hat bias within other areas of scientific inquiry in nutrition and this question warrants further investigation. Third, although our study (consistent with other recent reports)^{( 28 )} suggests that the food industry may manipulate science for self-interest, we present no data to suggest that industry-related researchers are intentionally complicit in this strategy. These limitations notwithstanding, our findings suggest that industry-related research hindered the pursuit of scientific truth about the health effects of SSB and may have harmed public health. Measures to mitigate these effects in future areas of scientific controversy within nutrition should be considered.