Belief beyond the evidence: using the proposed effect of breakfast on obesity to show 2 practices that distort scientific evidence^,

Abstract

Background: Various intentional and unintentional factors influence beliefs beyond what scientific evidence justifies. Two such factors are research lacking probative value (RLPV) and biased research reporting (BRR).

Objective: We investigated the prevalence of RLPV and BRR in research about the proposition that skipping breakfast causes weight gain, which is called the proposed effect of breakfast on obesity (PEBO) in this article.

Design: Studies related to the PEBO were synthesized by using a cumulative meta-analysis. Abstracts from these studies were also rated for the improper use of causal language and biased interpretations. In separate analyses, articles that cited an observational study about the PEBO were rated for the inappropriate use of causal language, and articles that cited a randomized controlled trial (RCT) about the PEBO were rated for misleadingly citing the RCT.

Results: The current body of scientific knowledge indicates that the PEBO is only presumed true. The observational literature on the PEBO has gratuitously established the association, but not the causal relation, between skipping breakfast and obesity (final cumulative meta-analysis P value <10⁻⁴²), which is evidence of RLPV. Four examples of BRR are evident in the PEBO literature as follows: 1) biased interpretation of one's own results, 2) improper use of causal language in describing one's own results, 3) misleadingly citing others’ results, and 4) improper use of causal language in citing others’ work.

Conclusions: The belief in the PEBO exceeds the strength of scientific evidence. The scientific record is distorted by RLPV and BRR. RLPV is a suboptimal use of collective scientific resources.

INTRODUCTION

Some beliefs about scientific topics are held true despite evidence refuting them, whereas other science-related beliefs are presumed true even though insufficient evidence exists to support or refute them (1). One such presumption is that regularly consuming compared with skipping breakfast protects against obesity or causes weight loss, which we refer to as the proposed effect of breakfast on obesity (PEBO)⁵ in this article. More precisely, the PEBO goes beyond indicating an association between skipping breakfast and obesity (in which breakfast consumption and lower weight are observed together) by indicating that the introduction of breakfast causes a decrease in body weight. We have chosen the PEBO as an example because 1) it is a topic that is amenable to conducting randomized controlled trials (RCTs), and thus, there are not insurmountable barriers to moving beyond observational evidence, and 2) it seems like a less politically charged topic than the topics of some previous publications on biased reporting [eg, sugar-sweetened beverages (2, 3) and breastfeeding (4)] to better facilitate the discussion of extrascientific factors that affect the fidelity of research reporting.

In this article, we first establish that the PEBO is only a presumption so that we can examine 2 factors that may influence the propensity to believe in a presumption beyond the available evidence. The first factor we identified is research lacking probative value (RLPV), which we defined as experiments or analyses that are 1) about questions that have already been sufficiently answered or 2) designed in such a way that they cannot advance the scientific knowledge about the question. As an example, when no association studies existed about the PEBO, the first few studies were certainly probative. Similarly, the first few studies to show the association in children compared with adults may be considered probative. However, at some point, additional similar observational analyses will only trivially add to our knowledge regarding the PEBO by studying nonmeaningfully different subgroups or by gratuitously replicating existing associations. We quantified RLPV by conducting a cumulative meta-analysis to show that the association, but not the causal relation, between breakfast and obesity has been more than sufficiently established.

The second factor was biased research reporting (BRR). We looked at 4 indications of BRR: 1) biased interpretation of one's own results, 2) improper use of causal language in describing one's own results, 3) misleadingly citing others’ results, and 4) improper use of causal language in citing others’ work. By discussing RLPV and BRR, we hope that 1) future efforts will be spent on elucidating novel associations and causal relations by conducting probative research, 2) future research results will be reported with greater fidelity by minimizing BRR, and 3) the belief in scientific topics will be grounded in scientific evidence.

METHODS

Establishing that the PEBO is a widely believed presumption

Lay-media, scientific, and government sources were searched for statements about breakfast and obesity to establish that the PEBO is widely believed. To evaluate our perception that the PEBO is only a presumption rather than an empirically supported scientific conclusion, scientific databases were searched for empirical human research about the PEBO. In addition, studies were reviewed from one published meta-analysis and 3 systematic reviews (5–8).

Cumulative meta-analysis to assess RLPV

We identified 92 unique articles about the PEBO that were cited in one published meta-analysis and 3 published systematic reviews (5–8). Briefly, Horikawa et al (5) meta-analyzed 19 studies of the association between breakfast consumption and odds of being overweight or obese in Asian and Pacific countries. Szajewska and Ruszczyński (8) systematically reviewed studies of children and adolescents in Europe and identified 16 studies. Mesas et al (6) systematically reviewed a number of eating behaviors related to obesity and reported 69 articles that looked at breakfast and obesity in English, Spanish, or Portuguese. Rampersaud et al (7) identified 16 articles in children and adolescents in a systematic manner, although the review was not declared a systematic review. All identified studies were observational. Studies were synthesized in a manner similar to the breakfast-obesity meta-analysis by Horikawa et al (5) as follows: breakfast consumption in each article had to be defined as a frequency (as opposed to investigating only the type or amount of breakfast consumed); the breakfast skipping group “was defined as the lowest category of breakfast frequency in an individual study” (5); body weight needed to be classified into overweight and/or obese; and analyses that adjusted for potential confounders were selected when available and appropriate. In addition, studies that considered both overweight and obese were included; mutually exclusive groups (eg, male and female subjects) were included as separate groups for analysis where possible and appropriate; and we limited the analysis only to full articles in the English language. SEs and ORs were calculated for each independent study group. In total, 58 of 92 studies fit these criteria with a total of 88 independent OR estimates. With the use of a random-effects model in Review Manager 5.1 software (The Nordic Cochrane Centre, The Cochrane Collaboration), we first synthesized all data by weighting each study group by the inverse variance of its point estimate. As a check, this method successfully reproduced the results of Horikawa et al (5) when limited only to data in their analysis (OR: 1.75; CI: 1.57, 1.95). Evidence of funnel-plot asymmetry was assessed graphically and by using Egger's linear regression (9). Funnel-plot asymmetry can be indicative of a publication bias or unaccounted heterogeneity in the analysis. Subsequently, we conducted a cumulative meta-analysis by sequentially synthesizing point estimates from the meta-analysis by publication year starting from 1994 and ending in 2011. All studies published within a given year were added concurrently.

Assessing BRR

For all ratings, AWB and MMBB independently rated the articles with input from DBA on the rating scales and criteria. All rating disagreements were settled by consensus between AWB and MMBB.

• 1) Biased interpretation of one's own results. Of the 92 articles identified for the cumulative meta-analysis, 88 articles contained abstracts indexed in PubMed in English. The results and conclusions stated in these abstracts were rated as follows: either not mentioning breakfast in relation to obesity (none) or indicating that breakfast was beneficial (positive), there was no relation between breakfast and obesity (no relation), breakfast was detrimental (negative), or the relation between breakfast and obesity was mixed (mixed). Although some studies implied mixed results (eg, breakfast was only stated as significant in one group and not mentioned in the other group), results were rated only on explicit mentioning of breakfast and obesity to allow reasonable tolerance for word limitations of abstracts.

• 2) Improper use of causal language in describing one's own results. The 88 rated abstracts were from observational studies; 76 studies were cross-sectional, 11 studies were longitudinal, and one study was a case-control study. Therefore, language concluding a finding of cause and effect was not appropriate within these 88 abstracts. Conclusions that included breakfast and obesity were subsequently rated causal if causal language was used or associative if the inference was limited to associations. In addition, conclusions were rated qualified causal if qualifiers such as “may” or “suggests” were included in the causal language because distinguishing between statements that used qualifiers to introduce a hypothesis compared with veiling a causal statement was subjective.

• 3) Misleadingly citing others’ results. We examined the manner in which articles cited an RCT reported by Schlundt et al (10). Schlundt et al (10) assigned women to a weight-loss plan that included or excluded breakfast and stratified them by prestudy habitual breakfast intake for a total of 4 study groups. They noted a P value <0.06 for the interaction of breakfast habit by breakfast assignment, with no significant main effect of breakfast consumption, which indicated that subjects who were assigned to change from their baseline breakfast frequency lost more weight than did subjects assigned to continue their baseline breakfast frequency (Table 1). We identified a total of 91 English-language articles that cited Schlundt et al (10) by searching the Web of Science (http://apps.webofknowledge.com) and Scopus (http://www.scopus.com) on 14 May 2012. Scopus contains citation records back to 1996; the Web of Science subscription for the University of Alabama at Birmingham is current from 1990 to year end 2011. We selected this article because 1) it was fairly well cited, and 2) the study had mixed results, which allowed us to potentially observe misleading citations both for and against breakfast. Articles were categorized on the basis of the way they cited Schlundt et al (10) as accurate, mildly misleading positive, explicitly misleading positive, mildly misleading negative, explicitly misleading negative, neutral, inaccurate unrelated, and otherwise unrelated. Positive was defined as misleadingly citing the results to make breakfast seem more beneficial, and negative was defined as misleadingly citing the results to make breakfast seem detrimental. Neutral meant that results were cited as indicating that breakfast and weight had no relation and did not mention the interaction trend, and the 2 unrelated categories indicated that breakfast and weight results were not cited (eg, the citation was related to other study aspects).

• 4) Improper use of causal language in citing others’ work. We examined to what extent authors extrapolated beyond the limitations of the study design when citing a National Weight Control Registry (NWCR) study by Wyatt et al (11). Wyatt et al (11) observed that “A large proportion of NWCR subjects (2313 or 78%) reported regularly eating breakfast every day of the week. Only 114 subjects (4%) reported never eating breakfast. There was no difference in reported energy intake between breakfast eaters and non-eaters, but breakfast eaters reported slightly more physical activity than non-breakfast eaters (p=0.05).” (11)

TABLE 1.

Weight-loss results from Schlundt et al (10)¹

	Baseline breakfast habits
	Eaters	Skippers	Total
Assignment (kg)
Breakfast	6.2 ± 3.3 [15]2	7.7 ± 3.3 [8]	6.73
No breakfast	8.9 ± 4.2 [14]	6.0 ± 3.9 [8]	7.8
Total	7.5	6.9	—

¹No main effects between the 2 factors were noted, but an interaction effect was noted at P < 0.06. Adapted with permission from reference 10.

²Mean ± SD; n in brackets (all such values).

³Mean (all such values).

This study was chosen because 1) it was fairly well cited [we identified a total of 91 articles that cited Wyatt et al (11) in English as described previously for Schlundt et al (10), of which 72 articles cited Wyatt et al (11) regarding the PEBO]; and 2) the NWCR describes a case series, which meant common behaviors observed within this population simply co-occurred with weight-loss maintenance, and therefore showed neither an association nor causation. Citations were rated as stating that the NWCR article concluded the relation between breakfast and obesity was causal or associative or correctly stated that the NWCR simply observed co-occurrence. The rest of the citations were rated as qualified associative or causal as previously described.

RESULTS

The PEBO is widely stated as true

We identified numerous statements and recommendations encouraging individuals to eat breakfast with the express purpose of influencing obesity. These recommendations came from popular health icons (12) and respected web-based health information outlets (13) and even the United States Surgeon General (14) (Table 2). Some statements included uncertainty (14), whereas other statements were hyperbolic and attributed unfounded behaviors to skipping breakfast [eg, “raiding the vending machine” (15)], and still other statements created specific temporal constraints on when breakfast must be consumed (16). A satellite symposium at Nutrition Society's 2012 summer meeting was even titled “How does breakfast help manage bodyweight?” (22) Although we cannot conclude that there is universal agreement in lay-media, government, and scientific-opinion leaders from these exemplars, we concluded that the PEBO is widely presumed true, as previously suggested (1).

TABLE 2.

Quotations that show breakfast is widely believed to protect against obesity

Quotation	Source
The fact is, when you're trying to lose body fat, you can't skip breakfast.	Dr Oz (12)
…there's ample evidence that the simple act of eating breakfast–every day–is a big part of losing weight, lots of weight.	WebMD (13)
Eating a healthy breakfast is a good way to start the day and may be important in achieving and maintaining a healthy weight.	US Surgeon General 14)
Skipping breakfast to lose weight makes you fatter - and far more likely to raid the vending machine.	Daily Mail (15)
Make sure to eat a healthy breakfast no later than 9 a.m., even if it means placing a bowl of oatmeal on your dressing table to eat while you put on your makeup.	Discovery Health (16)
Studies show that breakfast can be important in maintaining a healthy body weight.	Johns Hopkins Bloomberg School of Public Health (17)
With benefits of decreased body mass index, reduced fatigue, and improved cognition, we should all eat our Wheaties to start the day!	NYU Langone Internal Medicine Blog (18)
In fact, skipping breakfast actually increases your risk of obesity.	Mayo Clinic (19)
Research provides evidence that the simple act of eating breakfast every day is a big part of losing weight.	Scale Back Alabama (20)
Want to trim your waist? Try eating breakfast!	Academy of Nutrition and Dietetics (21)

Research does not clearly confirm or refute the PEBO

RCTs

There have been very few RCTs that investigated breakfast and weight change. RCTs related to the PEBO, with no consistent evidence that breakfast affects weight, are summarized in Table 3. One study even suggests that whether eating or skipping breakfast is more effective for weight loss depends on the individual's typical breakfast habits (10).

TABLE 3.

Randomized controlled trials that investigated breakfast interventions and obesity

First author, year of publication (reference)	Study length	Treatment description	Relevant results	Notes
Schlundt, 1992 (10)	12 wk	Skipping or eating breakfast stratified by baseline breakfast habits	Habitual breakfast eaters assigned to skip breakfast and habitual breakfast skippers assigned to eat breakfast lost more weight than subjects who maintained their breakfast habits; no main effect of breakfast on weight loss.	Interaction effect was P < 0.06
Geliebter, 2000 (23)	4 wk	Oatmeal, corn flakes, or water for breakfast	Water group lost 1.2 kg; breakfast groups did not change weight. No changes in lean and fat mass in groups.	Specific breakfast foods
Farshchi, 2005 (24)	2 wk	Controlled chronobiological study in which breakfast calories were shifted to lunch	No significant differences in weight, anthropometric measures, or body composition.	Controlled calories at subsequent eating occasions
Hirsch, 1975 (25)	1 wk/arm	All food consumed at dinner or at breakfast; crossover	Dinner-only resulted in a relative increase in weight compared with breakfast-only.	Not specifically breakfast compared with no breakfast
Crepinsek, 2006 (26)	3 school years	Universal-free school breakfast compared with standard School Breakfast Program	No difference in BMI or percentage of students overweight.	Not specifically breakfast compared with no breakfast
Powell, 1998 (27)	1 school year	Breakfast or a quarter of an orange	Breakfast intervention resulted in increased weight.	One-half of children were less than −1 SD weight-for-age; not specifically breakfast compared with no breakfast
Ask, 2006 (28)	4 mo	Free breakfast and supplements compared with control	Weight increased in control group and M subjects of intervention; BMI increased in control group but not in intervention.	Unclear if control and intervention groups were compared; one class room per treatment; not specifically breakfast compared with no breakfast
Rosado, 2008 (29)	12 wk	Ready-to-eat-cereal or habitual intake	No significant difference in weight between groups.	Included unrelated additional treatment groups; not specifically breakfast compared with no breakfast

Intermediate endpoint studies

Physiologic and behavioral mechanisms have been proposed to link breakfast and obesity, with the primary one being that breakfast consumption will result in a decreased cumulative energy intake by causing a compensatory hypophagia throughout the rest of the day. Although some studies showed a decreased energy intake at a subsequent eating occasion after eating compared with skipping breakfast, none of the studies showed a decrease in the sum of calories consumed across the study period when breakfast was eaten rather than skipped (Table 4). Other potential mechanisms have been proposed for the PEBO, such as chronobiology-dependent changes in metabolism; however, there is currently no clearly and consistently shown physiologic mechanism to explain the PEBO.

TABLE 4.

Studies that evaluated breakfast and energy intake

First author, year of publication (reference)	Study length	Treatment description	Relevant results	Notes
Astbury, 2011 (30)	1-d crossover	Breakfast or no breakfast; plus liquid preload before lunch	No significant differences in total energy intake (breakfast, preload, and ad libitum lunch)	Significant increased ad libitum test meal intake in no-breakfast group
Geliebter, 1999 (31)	1-d crossover	350 kcal oatmeal, sugared cornflakes, or water control	Liquid test meal intake was lower in the oatmeal but not cornflake conditions compared with water control	Overweight M and F subjects
Geliebter, 1999 (32)	1-d crossover	350 kcal oatmeal, sugared cornflakes, or water control	Liquid test meal intake was lower in the oatmeal but not cornflake conditions compared with water control	Lean and overweight M and F subjects; overweight subjects ingested even less after consumption of oatmeal
Kral, 2011 (33)	1-d crossover	Breakfast or no breakfast	Significantly higher energy intake in breakfast group	Similar intakes at lunch and rest of day
Leidy, 2011 (34)	1-d crossover	No breakfast, normal breakfast, or high-protein breakfast	No significant differences in total energy intake	Protein-rich breakfast lowered lunch intake compared with that with normal and no breakfast
1) Levitsky, 2013 (35)	1-d crossover	High carbohydrate, high fiber, or no breakfast; 1.4-MJ breakfasts	Significantly higher energy intake in breakfast groups (breakfast plus lunch)	Similar intakes at lunch
2) Levitsky, 2013 (35)	1-d crossover	Ad libitum or no breakfast	Significantly higher daily intake in breakfast group	Significant increase in lunch intake for no-breakfast group
Taylor, 2001(36)	2-d crossover	6 meals/d compared with 4 or 2 meals/d	No significant differences in total energy expenditure or intake	Wide range of BMI above overweight; F only

Observational evidence

Of the 88 abstracts used to investigate BRR, 86% of studies were cross-sectional. Longitudinal analyses used various measures of breakfast exposure, despite the relation between breakfast and obesity that potentially depends on how an individual defines breakfast eating (eg, number of days per week; including only certain foods) and obesity (eg, categorical obesity, continuous BMI, including or excluding overweight) (37) and were generally weak because of dataset limitations. There is a clear association between breakfast omission and excess weight as summarized in our cumulative meta-analysis (see RLPV below), and this association supports the PEBO but does not show causation.

RLPV

Studies included in the cumulative meta-analysis were conducted in 30 countries on 5 continents (Figure 1; countries in dark gray) with a variety of subpopulations (eg, rural and urban and men and women) and, thus, provide a widely generalizable association. Consistent with the general recommendations shown in Table 2 to eat breakfast with no regard for subpopulations, all OR estimates were pooled together regardless of the subpopulation. The meta-analysis of the 88 study groups resulted in a pooled OR of 1.55 (95% CI: 1.46, 1.65) (see Supplemental Figure S1 under “Supplemental data” in the online issue), with modest evidence of funnel-plot asymmetry as assessed graphically and by using Egger's linear regression (P = 0.0856) (9) (Figure 2). Such asymmetry can be indicative of a publication bias, unaccounted heterogeneity in the analysis, or asymmetry by chance.

FIGURE 1.

Map of countries where the proposed effect of breakfast on obesity has been studied. Dark gray denotes studies used in the meta-analysis or abstract analysis (n = 30). Black denotes studies used in the abstract analysis only (n = 6).

FIGURE 2.

Funnel plot of the meta-analysis. Dots represent each of 88 independent study group OR estimates. The vertical dashed line represents the meta-analyzed log OR. The solid line is an unweighted transformation of the Egger's linear regression test for funnel-plot asymmetry (9). The weighted intercept (0.5345) shows modest evidence of funnel-plot asymmetry (P = 0.0856), which may represent a publication bias, unaccounted heterogeneity in the meta-analysis, or asymmetry by chance.

The cumulative association between breakfast and obesity was significant at α = 0.05 (2-tailed test) after the first year, which included only 3 study groups, and remained so throughout the entire cumulative time course (Figure 3). Of particular note was that, by 1998, after synthesizing only 5 estimates, the P value was <0.001, which Sterne and Smith proposed to be interpreted as strong evidence of a relation (38). By 2011, the P value was <10⁻⁴². Although no universal threshold exists for what constitutes sufficient observational evidence, we posit that 10⁻⁴² is excessive, and therefore, it was gratuitous to conduct additional association studies regarding the PEBO well before 2011.

FIGURE 3.

Cumulative meta-analysis of studies analyzed in the meta-analysis. Each data point represents the synthesis of all studies included year on year (n = 58 studies with 88 independent groups). Diamonds represent the ORs of skipping breakfast and being overweight or obese bounded by a 95% CI. The black horizontal line is set at an OR of 1. The final OR is 1.55 (95% CI: 1.46, 1.65). Squares represent P values of ORs for each year of the analysis. Two horizontal dotted lines represent the traditional P = 0.05 significance and P = 0.001, which was suggested to represent strong evidence of an association (38). The final P value is 10⁻⁴².

Evidence of BRR in abstracts and citing behaviors

Of our 4 examples of BRR, the first 2 examples represented infidelity in the reporting of one's own work, whereas the second 2 examples represented the distortion of others’ research.

Biased interpretation of one's own results

If all 88 of the rated abstracts accurately reported results and conclusions, and all of these articles only investigated breakfast and obesity, all counts should appear on the diagonal in Figure 4, which was not the case (ie, all diagonal cells should be black). In particular, 65% of studies that reported positive results for breakfast also mentioned the results in the abstract's conclusion compared with only 39% of studies with nonpositive results (2-proportion test: P = 0.0492). Thus, a disproportionate number of abstracts with nonpositive results failed to include a nonpositive conclusion. Note that none of the studies that reported positive results had conclusions categorized as no relation, negative, or mixed, whereas one abstract that did not mention breakfast and obesity in the results still had a positive conclusion toward the PEBO.

FIGURE 4.

Concordance between results and conclusions from 88 identified abstracts. Abstracts were rated on the basis of whether the results or conclusions stated that eating breakfast was associated with lower obesity (Positive), showed no relation between breakfast and obesity (No Relation), showed that eating breakfast was associated with higher obesity (Negative), showed mixed relations (Mixed), or did not mention an analysis between breakfast and obesity (None). Cell shading represents the percentage of conclusion ratings within a result-rating column, with black representing 100% and white representing 0%. Marginal counts are not shaded.

Improper use of causal language to describe one's own results

Language that concluded a finding of cause and effect was not appropriate for this set of 88 abstracts from observational studies. However, of the 42 abstracts that included conclusions about breakfast and obesity, 11 abstracts made unqualified causal claims about breakfast consumption and obesity, whereas an additional 10 abstracts used qualifiers to introduce uncertainty into the causal statements; 21 articles limited the abstract conclusions to noncausal language (Figure 5). Thus, 26–50% of studies were ascribed greater inferential strength than the study designs warranted.

FIGURE 5.

Authors’ use of causative language in their own observational studies. The left pie chart shows that 48% (n = 42) of 88 abstracts made conclusions about breakfast and weight, which is broken down by the use of causal language in the right pie chart.

Misleadingly citing others’ results

Of the 91 articles that cited Schlundt et al (10) , 46 articles cited the article with respect to the PEBO, whereas the other citations were related to other endpoints (eg, snacking frequency) or were otherwise unrelated (Figure 6). Of the PEBO-relevant citations, only 17% of articles cited the results accurately. Of the 29 articles that misleadingly cited Schlundt et al (10), 28 articles mildly or explicitly cited Schlundt et al (10) positively toward breakfast with only 1 article that cited Schlundt et al (10) negatively toward breakfast (P = 1.12 × 10^–7; 2-sided binomial test against H₀: P-positive|misleading citation = 0.5). The only article (2%) that cited the results misleadingly negatively toward breakfast also cited the results accurately elsewhere in the article (44). These results show that a sizeable number of citations of Schlundt et al (10) were misleading (62% of the PEBO-relevant citations), and they were almost exclusively biased in favor of breakfast.

FIGURE 6.

Categorization of 91 articles that cited Schlundt et al (10). Articles that cited Schlundt et al (10) were categorized as shown in the table inset in the figure. All articles were included in the left pie chart, with only the relevant citations presented in the right pie chart (n = 42). ¹The one study that was explicitly misleadingly negative also cited Schlundt et al (10) accurately elsewhere in the article.

Improper use of causal language in citing others’ work

Of the 91 articles that cited Wyatt et al (11), 72 articles cited the article with respect to the PEBO. Of these articles, 29% correctly described the relation between breakfast and weight-loss maintenance as simply co-occurring, whereas 26% stated the 2 were associated, and 22% made statements that causally linked breakfast and obesity (Figure 7). The rest of the articles were rated qualified associative or causal (4% and 18% of relevant abstracts, respectively). Thus, 48% of the PEBO-relevant citations of Wyatt et al (11) explicitly ascribed a stronger inferential meaning to the article than was warranted, with an additional 22% of articles that potentially did so depending on the interpretation.

FIGURE 7.

The use of causal language in 91 articles that cited Wyatt et al (11). Articles that cited Wyatt et al (11) were categorized as shown in the table inset in the figure. The left pie chart represents all articles that cited Wyatt et al (11), whereas the right pie chart was limited to relevant citations (n = 72). ¹Unratable citations include accurate citations unrelated to breakfast and weight and citations for which it was unclear what was being attributed to the Wyatt et al (11) article.

DISCUSSION

We have shown that 1) the PEBO is presumed and stated as true despite equivocal evidence; 2) the gratuitous replication of associations between breakfast and obesity showed that numerous nonprobative studies exist in the PEBO literature; and 3) there is evidence of a bias with respect to the reporting of one's own and others’ research.

We reiterate that we used breakfast and obesity as an example for RLPV and BRR and did not consider other important associations with breakfast, such as cognitive function, or other important associations with obesity, such as fruit and vegetable consumption. We also acknowledge that our analysis may not have been fully comprehensive because we selected studies on the basis of previous research syntheses from which we could calculate ORs. However, additional studies that might have been identified from a de novo systematic review were unlikely to meaningfully affect the final P value of the cumulative meta-analysis because of its magnitude and only weak evidence of a publication bias. Therefore, additional unidentified studies would have likely contributed to our conclusion that the association has been excessively established.

Our cumulative meta-analysis that showed the presence of RLPV had a final P value <10⁻⁴², which was almost forty degrees of magnitude less than the proposed significance level of Sterne and Smith (38). Despite this gratuitous replication of effort, we are still uncertain of the role that breakfast plays in obesity because these observational studies informed association and not necessarily causation. Each study and analysis that is conducted costs time and resources that could be dedicated to unveiling novel associations or engaging in randomized trials that can better define causal relations. Therefore, the evidence of RLPV is also evidence of an unproductive use of scientific resources.

We do not suggest that replication be suppressed (55); rather we want to discourage gratuitous replication. Some journals, eg, now require bullet points that identify what is already known compared with what is needed. Although this may help stifle the publication of RLPV, the research has already consumed resources if a manuscript is being written. Unfortunately, there is an obligation to make nonprobative replications publicly available, regardless of the outcome, lest we encourage publication bias. Perhaps a more useful exercise would be for researchers to better reflect on the body of evidence before engaging in research. This exercise could be facilitated by research registration that is subjected to peer review (56).

Another means of increasing the marginal probative value of a study relative to an established body of research evidence is to conduct experiments that directly test the relation of interest rather than creating causal extrapolations from observational or intermediate-endpoint studies in obesity (57). These experiments can often be conducted with established designs, are generally cost effective, and often have relatively low ethical concern, particularly when studying nutrition. However, the United States and WHO clinical trial registries have only 1 study registered to test the PEBO (as of 9 October 2012; www.clinicaltrials.gov; NCT01781780). Researchers with the Nutrition Obesity Research Center at the University of Alabama at Birmingham, in collaboration with other research centers, are conducting an effectiveness study that will test recommendations to eat breakfast or not to eat breakfast, as well as having a control group, over 16 wk at 5 sites, with groups stratified by typical breakfast-consumption habits. Not including personnel costs, the study is estimated to cost in the neighborhood of $40,000. This study shows the feasibility of conducting randomized controlled trials on obesity-targeted population health recommendations rather than relying on observational extrapolations.

It is unclear why BRR and RLPV are present in the PEBO literature. A hypothetical framework to tie these factors together is shown in Figure 8. Frequent assertions of the general benefits of consuming breakfast, most easily summarized as “breakfast is the most important meal of the day,” could potentially predispose individuals to ascribe other positive attributes to breakfast because of the “halo-effect,” in which some foods are considered to be intrinsically good and wholesome (58). In addition, the repeated exposure to RLPV may further predispose researchers to specifically believe in the PEBO because of the “mere exposure effect”: individuals tend to believe something to which they have been repeatedly exposed (59). This predisposition may be exacerbated if the exposure consists of biased information that communicates stronger conclusions than the evidence warrants, as shown in BRR. In turn, individuals may be less likely to seek out or accept information contrary to their preconceptions (confirmation bias) as a way to decrease the mental discomfort associated with conflicting information (cognitive dissonance) (60). A resultant biased mental repository of information would be consistent with why individuals believe more strongly in a presumption than the existing evidence objectively supports (a hypothetical representation is shown in Figure 8D). Conversely, “white hat bias” (2) suggests the influence of bias on research distortion may be more direct whereby individuals (consciously or unconsciously) might ignore the weakness of the scientific evidence regarding a presumption in favor of believing in the wholesome nature of the exposure (ie, breakfast) in hopes of addressing a formidable consequence (ie, obesity). Although we could not directly test either framework in the current study, we hypothesize that these factors may influence the propensity of authors to bias research reporting.

FIGURE 8.

Hypothetical model of how BRR and RLPV may be involved in perpetuating presumptions. A: Exposure to such phrases as “breakfast is the most important meal of the day” may predispose individuals to believe positive things about breakfast because of the “halo effect” (58). B: The RLPV related to the PEBO may augment this predisposition to believe the PEBO through the “mere exposure effect,” (59) particularly when the research is presented in a biased manner (eg, in the presence of BRR). C: Individuals tend to seek out information confirming their point of view and reject information to the contrary (confirmation bias) to prevent or reduce cognitive dissonance (60), thereby retaining a biased sample of information. D. A hypothetical illustrative graph of the comparison of strength of conviction compared with the strength of existing evidence supporting the PEBO. E: Together, these cognitive biases may predispose researchers to bias their research reporting, which, thereby, would feed the cycle. BRR, biased research reporting; PEBO, proposed effect of breakfast on obesity; RLPV, research lacking probative value.

Beyond the investigator-centric framework proposed in Figure 8, the implications of BRR extend to other evidence consumers. When research reporting is biased, the integrity of scientific information deteriorates because it drifts from the original source, much like the childhood game of “telephone.” If the fidelity of the scientific message cannot be replicated from the results to conclusions within a study [(as shown in the first and second examples of BRR in the current article and “spin” by Boutron et al (61)], the reporting of the scientific finding is distorted before it even reaches press releases (62). Therefore, evidence consumers are receiving biased information.

Ironically, raising awareness of a bias often helps individuals identify biases in others rather than oneself, which is a so-called “bias blind spot” (63). Although we hope that the results presented in the current article will help individuals identify biased reporting in their own work (ourselves included), it is the peer-review process that may be best improved by raising awareness of bias. As writers and reviewers, we need to be vigilant for lapses in objectivity and for studies of nonprobative value to hopefully keep scientific beliefs grounded in scientific evidence.