Abstract
Objective
To systematically review the conduct and reporting of formula trials.
Design
Systematic review.
Data sources
Medline, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) were searched from 1 January 2006 to 31 December 2020.
Review methods
Intervention trials comparing at least two formula products in children less than three years of age were included, but not trials of human breast milk or fortifiers of breast milk. Data were extracted in duplicate and primary outcome data were synthesised for meta-analysis with a random effects model weighted by the inverse variance method. Risk of bias was evaluated with Cochrane risk of bias version 2.0, and risk of undermining breastfeeding was evaluated according to published consensus guidance. Primary outcomes of the trials included in the systematic review were identified from clinical trial registries, protocols, or trial publications.
Results
22 201 titles were screened and 307 trials were identified that were published between 2006 and 2020, of which 73 (24%) trials in 13 197 children were prospectively registered. Another 111 unpublished but registered trials in 17 411 children were identified. Detailed analysis was undertaken for 125 trials (23 757 children) published since 2015. Seventeen (14%) of these recently published trials were conducted independently of formula companies, 26 (21%) were prospectively registered with a clear aim and primary outcome, and authors or sponsors shared prospective protocols for 11 (9%) trials. Risk of bias was low in five (4%) and high in 100 (80%) recently published trials, mainly because of inappropriate exclusions from analysis and selective reporting. For 68 recently published superiority trials, a pooled standardised mean difference of 0.51 (range −0.43 to 3.29) was calculated with an asymmetrical funnel plot (Egger’s test P<0.001), which reduced to 0.19 after correction for asymmetry. Primary outcomes were reported by authors as favourable in 86 (69%) trials, and 115 (92%) abstract conclusions were favourable. One of 38 (3%) trials in partially breastfed infants reported adequate support for breastfeeding and 14 of 87 (16%) trials in non-breastfed infants confirmed the decision not to breastfeed was firmly established before enrolment in the trial.
Conclusions
The results show that formula trials lack independence or transparency, and published outcomes are biased by selective reporting.
Systematic review registration
PROSPERO 2018 CRD42018091928.
Introduction
Breast milk substitutes, commonly termed formula milks, are consumed by most infants in North America and Europe, and global sales rose from 3.5 kg per child in 2005 to 7.4 kg per child in 2019.1 Formula milk is very rarely medically indicated, but the social changes of industrialisation and aggressive marketing of formula products have contributed to the cultural normalisation of formula feeding in many regions.2 Use of formula in place of breastfeeding is associated with short and long term health risks, from burns, colic, and constipation to pneumonia, developmental effects, and fatal diarrhoea.3 Formula is sometimes a young infant’s only source of nutrition, and can be consumed at very high levels, up to 20% of an infant’s body weight per day, during a sensitive period of development.3 Clinical trials are therefore required by regulators to show the safety of new formula products, and trials are also used by manufacturers to support marketing claims.4 The widespread use of formula products, their associated health risks, and the high individual levels of intake during early development highlight the importance of the evidence generated from formula milk trials. Associations for the nutrition industry cite their investment in world class research and development, but several groups from academia and regulatory science have raised concerns about the conduct and reporting of formula milk trials.3 5 6 Some have suggested that trial procedures might contravene the International Code of Marketing of Breast-milk Substitutes, for example, by providing free formula to participants.7 8 We therefore conducted a systematic review to evaluate the conduct and reporting of formula milk trials. We were specifically interested in understanding the risk of bias in published formula trials and if trial procedures could cause harm by undermining breastfeeding of participants.
Methods
This systematic review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).9 We searched Medline, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) for clinical trials comparing two or more different formula products in children aged <3 years, published between 1 January 2015 and 31 December 2020 (the supplementary material describes the search strategy). We also identified all trials registered on the International Clinical Trials Registry Platform from when registration of clinical trials was made mandatory, and the status of trial registration of all trials published since 1 January 2006, to evaluate the risk of selective reporting at the trial level.10 Prospective trial registration was defined as registration of the trial in a clinical trial registry approved by the World Health Organization before, or up to one month after, the reported start of the study. We excluded trials of supplements given separately to formula, breast milk or fortifiers of breast milk, and trials comparing the frequency or volume of consumption of formula without changing the ingredients or formulation. The primary outcome of the systematic review was overall risk of bias for the trial primary outcome, evaluated with Cochrane risk of bias version 2.0 (ROB2).11 Secondary outcomes were risk of bias in individual domains of ROB2, how favourable the primary outcomes and conclusions of the trial were, measures of selective reporting at the trial level, and risk of trial procedures undermining breastfeeding. Selection of studies, extraction of data, and coding of outcomes were extensively piloted and carried out by at least two researchers. Disagreements were resolved by rechecking the original articles and discussion leading to a consensus decision.
To identify the primary outcome of each trial, we used the first primary outcome listed in the record of a clinical trial registry approved by the World Health Organization or, if not registered, we used the methods section or first reported result in the main trial manuscript. Risk of undermining breastfeeding and clarity of describing the trial interventions were evaluated with criteria from recent Delphi consensus guidance on the conduct and reporting of formula trials.12 The extent to which the outcomes and conclusions were favourable was evaluated as previously described.13 For statistical analysis of the primary outcomes of the trial, effects were estimated by calculating standardised mean difference for continuous outcomes and odds ratios for dichotomous outcomes; odds ratios were then transformed into standardised mean difference, as previously described.14 Unreported means and standard deviations were computed from available raw data or estimated from interquartile range and median values.15
Publication bias was evaluated with contour enhanced funnel plots and the trim-and-fill procedure, and by summarising unpublished formula trials and abstract publications.16 We recorded the source of funding and conflicts of interest of the authors, as declared in published reports, and evaluated conflicts of interest as low where the trial funder did not have a commercial interest in the outcome of the trial and no study authors had a financial link to an organisation with a commercial interest in the outcome of the trial. We analysed associations between the characteristics of the trial, including the aim of the trial, age group, geographical location (classified according to the human development index), size, and publication year, and risk of bias for each trial, support for breastfeeding, and outcomes of the trial.
Statistical analyses with Review Manager 5.3 and R 4.0.2 were conducted separately for superiority trials and equivalence or non-inferiority trials. Superiority trials are randomised controlled trials designed to show that the intervention is better than the comparator, non-inferiority trials are designed to show that the intervention is no worse than the comparator, and equivalence trials are designed to show that no meaningful difference exists between the intervention and the comparator. Random effects meta-regression was used to evaluate associations between standardised mean difference and features of the trial. High heterogeneity was expected because of the diverse features of the trials, and therefore we used the Hartung-Knapp adjustment for random effects model weighted by the inverse variance method. Heterogeneity was assessed by τ2 with the Sidik-Jonkman estimator. The protocol of this systematic review was registered at PROSPERO after piloting the processes for selection of studies, extraction of data, and assessment of risk of bias but before piloting of the analysis of the data.17
Patient and public involvement
Patients or the public were not directly involved in this study, but contributed to development of the Delphi consensus guidance which we used to evaluate the risk of undermining breastfeeding and reporting the use of other nutritional intake in formula trials.12
Results
Our search results are summarised in the PRISMA flowchart (supplementary fig 1). After screening the title, abstract, and full text, we found 307 eligible main trial publications since 2006, the first calendar year after registration of clinical trials was made mandatory.10 We summarised the status of trial registration for all 307 trials published since 1 January 2006, and undertook a more detailed evaluation of 125 trials published since 1 January 2015. Of the 307 main trial publications identified since 2006, 149 (49%) had a record of trial registration, of which 73 (24%) were registered prospectively, evaluating 13 197 children (supplementary table 1). We identified another 111 unpublished formula trials in 17 411 children registered since 2006, and 64 of 85 (75%) trials that reported trial status had completed the trial (supplementary table 2).
Characteristics of recently published formula milk trials
Between 2015 and 2020, we identified 125 published trials evaluating 23 757 randomised children (supplementary table 3). Six more trials were published as abstracts only and were not analysed further (supplementary table 4). The recently published full text formula trials were mainly conducted in infants (90%, supplementary table 3), and were mainly carried out in Europe (42%), Asia (28%), or North America (18%). Trials were small (median sample size 114, interquartile range 60-204) and usually had a superiority aim (76% of trials where the aim was clear). Formula interventions studied were commonly prebiotics or probiotics, alone or combined with other ingredients (42%), changes to protein source or content (29%), or fat content (12%). The most common primary outcome was weight gain (36%), measured in half of these trials according to regulatory guidance for establishing adequate growth in infants exclusively fed formula milk.18 Other common primary outcomes were intestinal health (26%), including faecal analysis and gastrointestinal symptoms, measures of nutrient absorption (10%), behaviour (6%), and allergy (6%).
Transparency of aims in recently published formula trials
Between 2015 and 2020, 57 (46%) of 125 recently published trials had a prospective registration record and two trials had a prospective publicly available protocol (table 1 and supplementary table 5). Another 33 (26%) trials had a retrospective trial registration record, dated a median of 24 (interquartile range 7-48) months after the start of the study. A clear study aim was described in 79 (63%) trials, and this proportion was similar for registered and unregistered trials. A clear primary outcome was described in 59 (47%) trials and was more common in prospectively registered trials (60%) than unregistered trials (30%). Protocols dated before the start of the study were only available for 11 (9%) trials, of which eight had a clear study aim and clear primary outcome (table 1).
Table 1.
Transparency of aims in formula trials published between 2015 and 2020
| Trials | Total | |||
|---|---|---|---|---|
| Prospectively registered | Non-prospectively registered | Unregistered | ||
| Issues with definition of study aim: | ||||
| Unclear which arm is control group, or whether aim is superiority, equivalence, or non-inferiority | 12/57 (21) | 6/35 (17) | 8/33 (24) | 26/125 (21) |
| Dataset inappropriate for stated aim of trial | 9/57 (16) | 6/35 (17) | 5/33 (15) | 20/125 (16) |
| Clear study aim | 36/57 (63) | 23/35 (66) | 20/33 (61) | 79/125 (63) |
| Issues with description of primary outcome: | ||||
| Multiple primary outcomes or unclear description of primary outcome | 12/57 (21) | 7/35 (20) | 16/33 (48) | 35/125 (28) |
| Timing or measurement method for evaluating primary outcome unclear | 11/57 (19) | 13/35 (37) | 7/33 (21) | 31/125 (25) |
| Clear primary outcome | 34/57 (60) | 15/35 (43) | 10/33 (30) | 59/125 (47) |
| Both study aim and primary outcome clearly described | 26/57 (46) | 10/35 (29) | 7/33 (21) | 43/125 (34) |
| Protocol availability: | ||||
| Prospective protocol publicly available, or provided by author or funder on request | 9/57 (16) | 2/35 (6) | 0/33 (0) | 11/125 (9) |
| Retrospective or undated protocol publicly available, or provided by author or funder on request | 10/57 (18) | 9/35 (26) | 0/33 (0) | 19/125 (15) |
| Author or funder declined to provide protocol | 20/57 (35) | 5/35 (14) | 2/33 (6) | 27/125 (22) |
| Author or funder responded but did not provide protocol | 5/57 (9) | 8/35 (23) | 6/33 (18) | 19/125 (15) |
| No response from author or funder and no publicly available protocol | 13/57 (23) | 11/35 (31) | 25/33 (76) | 49/125 (39) |
| Prospective protocol and clear study aim and clear primary outcome | 8/57 (14) | 0/35 (0) | 0/35 (0) | 8/125 (6) |
Values are No of trials/total No of trials (%).
Prospective trial registration was defined as trial registration in a clinical trial registry approved by the World Health Organization before, or up to one month after, the reported start of the study. Corresponding authors were contacted to request trial protocols and statistical analysis plans up to three times. Only four statistical analysis plans were received: one undated, three dated, two to 13 months after completion of the study.
Evidence of publication and reporting bias in recently published formula trials
In formula trials published between 2015 and 2020, analysis of the primary outcome was favourable in 53 (42%) trials, the author’s interpretation of the outcome was favourable in 86 (69%) trials, and the abstract conclusions were favourable in 115 (92%) trials (supplementary table 6). For 68 published superiority trials, the standardised mean difference favoured the intervention formula in 57 (84%) trials, the control formula in eight (12%), and neither product in three (4%) trials. For superiority trials, we calculated the pooled standardised mean difference (0.51, 95% confidence interval 0.34 to 0.68, I2=84%, range −0.43 to 3.29; fig 1) with an asymmetrical funnel plot (fig 2 (top); Egger’s test P<0.001) and found a reduced effect size after correction for asymmetry (fig 2 (bottom); adjusted standardised mean difference 0.19, −0.03 to 0.41). For 34 published non-inferiority or equivalence trials, we calculated a pooled standardised mean difference of −0.06 (−0.15 to 0.02, I2=44%, range −1.04 to 0.42; supplementary fig 2). For 23 published trials designed according to regulatory guidance for demonstrating adequate growth, the authors interpreted the primary outcome as favourable in 22 (96%) trials. Only 12 of 23 (52%) trials showed equivalence with a two sided 95% confidence interval (or 15 (65%) trials with a 90% confidence interval), using the American Academy of Pediatrics recommended equivalence margin of a difference in weight gain of 3 g/day (fig 3).18
Fig 1.
Primary outcomes for formula trials with a superiority aim published between 2015 and 2020. Standardised mean difference and 95% confidence intervals for the primary outcome of the 68 formula trials designed with the aim of showing that the intervention was better than the comparator, where standardised mean difference could be calculated
Fig 2.
Funnel plot for formula trials with a superiority aim published between 2015 and 2020. Effect size, expressed as standardised mean difference, is shown as a purple dot for each trial primary outcome, against the corresponding standard errors. Colours indicate regions of statistical significance (top panel). Unadjusted pooled standardised mean difference is 0.51 (0.34 to 0.68), Egger’s test for funnel plot asymmetry=4.111, P<0.001. White dots represent standardised mean difference imputed with Duval and Tweedie’s trim-and-fill procedure until funnel plot symmetry is reached (bottom panel). Adjusted standardised mean difference is 0.19 (−0.03 to 0.41)
Fig 3.
Primary outcomes for formula trials designed to show adequate infant growth, published between 2015 and 2020. Data are mean differences (95% confidence intervals) and mean differences (90% confidence intervals) in weight gain between intervention and control formula during the intervention period, for the 23 trials that were designed according to guidance of Food and Drug Administration for demonstrating adequate growth. Equivalence margins of 3 g/day weight gain are indicated by dotted lines. Authors interpreted the primary outcome as favourable in 22 (96%) trials
Cochrane risk of bias assessment of recently published formula trials
For trials published between 2015 and 2020, risk of bias was low in five (4%) and high in 100 (80%) trials (table 2, supplementary table 7). In domain 1, the randomisation process, four (3%) trials were at high risk of bias because of inadequate concealment of allocation. In domain 2, deviations from the intended interventions, 52 (42%) trials were at high risk: 32 trials that evaluated adherence, mainly because the trials used a simple per protocol analysis which excluded ≥10% of participants (median 30%, interquartile range 15-45%) without accounting for missing data in the analysis; and 20 trials that evaluated assignment, mainly because of inappropriate exclusion of ≥10% of participants from the analysis (23%, 18-34%) for reasons such as adverse events or non-adherence. In domain 3, missing outcome data, 18 (14%) trials were at high risk of bias because of ≥10% missing outcome data (35%, 23-42%) with no evidence that the estimate was stable with methods such as multiple imputation, and a high likelihood that missingness in the outcome data depended on their true value. In domain 4, outcome measurement, five (4%) trials were at high risk of bias because assessors of outcome were not blinded and the outcome was subjective. In domain 5, selective reporting, 16 (13%) trials were at high risk of bias, most commonly because of analysis of an inappropriate dataset for the stated aim, for example analysis of an intention to treat dataset for an equivalence aim without presenting the per protocol results. In 83 (66%) trials, there were some concerns about selective reporting because prospective trial registration was absent, multiple primary outcomes or evaluation methods were registered, methods for measurement or analysis of primary outcomes were poorly specified, or reporting was incomplete.
Table 2.
Risk of bias in formula trials published between 2015 and 2020
| Randomisation process | Deviation from interventions | Missing outcome data | Outcome measurement | Selective reporting | Overall risk of bias | |
|---|---|---|---|---|---|---|
| Low | 103/125 (82) | 36/125 (29) | 67/125 (54) | 107/125 (86) | 26/125 (21) | 5/125 (4) |
| Some concerns | 18/125 (14) | 37/125 (30) | 40/125 (32) | 13/125 (10) | 83/125 (66) | 20/125 (16) |
| High | 4/125 (3) | 52/125 (42) | 18/125 (14) | 5/125 (4) | 16/125 (13) | 100/125 (80) |
Values are No of trials/total No of trials (%).
Risk of bias was evaluated with Cochrane risk of bias version 2.0 (ROB2).9
Conflict of interest in recently published formula trials
Eight (6%) trials published between 2015 and 2020 did not report a funding source. Of 117 trials that reported sources of funding, 98 (84%) received support from the formula milk industry, although in seven trials funding was limited to donating the study formula (table 3; supplementary table 8). Seventy five of 98 (77%) trials supported by the formula industry had at least one author affiliated to a formula company, and the formula industry was involved in the statistical analysis or writing of the trial report, or both, in 60 of 81 (74%) trials where their role was clearly described. Eighty seven of 112 (78%) trials that reported conflicts of interest of authors had at least one author who had a conflict of interest related to the formula industry. Overall, 17 (14%) published trials had a low level of conflicts of interest according to our a priori definition that the main source of funding had no commercial interest in the outcome of the trial and all of the authors of the study declared no financial ties to an entity with a commercial interest in the outcome of the trial.
Table 3.
Conflict of interest in formula trials published between 2015 and 2020
| Independent funding (n=19)* | Formula donated by formula industry but no other industry support reported (n=7) | Formula industry funding reported (n=91) | Total | |
|---|---|---|---|---|
| At least one author reported a conflict of interest related to formula industry | 2/16 (13) | 4/7 (57) | 81/89 (91) | 87/112 (78) |
| At least one author affiliated to formula industry | 0/19 | 2/7 (29) | 73/91 (80) | 75/117 (64) |
| Formula industry sponsor involved in statistical analysis or trial reporting | — | 1/6 (17) | 59/75 (79) | 60/81 (74) |
Values are No of trials/total No of trials (%).
Independent funding means a funding source not related to the formula industry (n=17) or authors reported that the study did not have any formal funding (n=2). Denominators vary because of lack of reporting for some variables. Source of funding was not reported for eight studies, author conflict of interest was not reported for nine studies, and role of formula sponsor in statistical analysis or trial reporting was not reported for 17 studies.
Support for breastfeeding evaluated with international guidance for formula trials
We evaluated reported support for breastfeeding in formula trials published between 2015 and 2020 according to criteria derived from consensus guidance for the conduct and reporting of formula trials.12 Eighty four (67%) trials clearly described if participants were receiving breast milk at enrolment (supplementary table 9). In 33 of 38 (87%) trials where some infants were likely to be receiving breast milk at enrolment, the International Code of Marketing of Breast-milk Substitutes seemed to be contravened by providing free formula to participants in the trial, which would not have been provided outside of the trial (supplementary table 9). In five of these, participants were required to consume a minimum or fixed daily volume of formula, which might have undermined breastfeeding.19 20 21 22 23 In 73 of 87 (84%) trials in infants fully formula fed, whether the decision not to breastfeed was firmly established before enrolment was unclear, and 18 of these trials enrolled infants in the first week of life. Overall, in our evaluation, we found that 15 (12%) trials were at low risk of undermining breastfeeding, 45 (36%) trials were at high risk, and 65 (52%) had some concerns (table 4). Trials also failed to adhere to other recommendations for reporting nutritional intakes in formula trials. Insufficient information was available to link the products used in the trials with marketed formula products in 101 (81%) recent trials (supplementary table 10), and 53 (42%) trials did not provide sufficient information to confirm that the composition of the control formula conformed to international or local standards. In 93 (74%) trials, no information was reported about length of breastfeeding in the study participants, before or during the trial, and in 100 of 115 (87%) trials where non-milk feeds could be relevant, we found no information about timing of introduction of complementary food or intake of other relevant nutritional products.
Table 4.
Risk that breastfeeding was undermined in formula trials published between 2015 and 2020
| Risk | Receiving breast milk at enrolment | Most common reason | No breast milk at enrolment | Most common reason | Total |
|---|---|---|---|---|---|
| Low | 1/38 (3) | Good breastfeeding support. Free formula during inpatient stay but likely consistent with local practice | 14/87 (16) | No breastfeeding for a substantial length of time before enrolment* | 15/125 (12) |
| Some concerns | 11/38 (29) | Free formula which was not available outside of the trial context | 55/87 (63) | No information to judge if the decision not to breastfeed was firmly established. Enrolment occurred after the first week of life | 65/125 (52) |
| High | 26/38 (68) | Free formula and no statement that breastfeeding was supported | 18/87 (21) | No information to judge if the decision not to breastfeed was firmly established. Enrolment occurred in the first week of life | 45/125 (36) |
Values are No of trials/total No of trials (%).
Three trials enrolled infants fed only formula or not breastfed for ≥3 days before enrolment; one trial ≥5 days; and eight trials between ≥2 weeks and ≥2 months. We judged risk of undermining breastfeeding as low in trials where children did not receive breast milk at enrolment, if the publication made it clear the decision not to breastfeed was firmly established before enrolment, for example an eligibility criterion of no breast milk for ≥3 days.10 For trials with some breastfeeding at enrolment, we judged risk of undermining breastfeeding as low if all Delphi consensus criteria were adhered to, and high if no criterion was adhered to.10
Trial features associated with risk of bias and favourability of outcomes
Supplementary tables 11-16 and supplementary figs 3-6 show the relations between the characteristics of the trials and risk of bias, outcomes of the trials, or support for breastfeeding. All trials with a low risk of bias had a superiority aim, and a low risk of bias was associated with low conflicts of interest (odds ratio 11.36, 95% confidence interval 1.74 to 73.98). Low risk of bias related to measurement of outcome (P<0.001) and selective reporting (P=0.005), but not overall low risk of bias, was associated with a higher human development index. For superiority trials, effect size was larger in trials with a high risk of bias (P=0.03) and smaller in trials with a low risk of bias (P=0.03), but was not related to the level of commercial involvement in the trial. Favourable trial conclusions were associated with a high risk of bias (odds ratio 7.58, 95% confidence interval 1.95 to 29.46) and lower human development index (P=0.005). We found no association between effect size or favourable conclusions and publication delay. We did not find any relation between the characteristics of the trials and support for breastfeeding, or any changes in risk of bias, favourable conclusions, or support for breastfeeding over time.
Discussion
Principal findings
In this systematic evaluation of formula milk trials, we found an almost universal lack of transparency, and evidence of selective reporting between and within trials. The widespread and increasing use of formula in children at a sensitive period of development emphasises the importance of scientific rigour in this area of clinical investigation. Our findings suggest that recent formula trials lack that scientific rigour, and published outcomes are biased by selective reporting.
We found more registered unpublished trials than prospectively registered published trials. Few of the recent published trials had a clear a priori registered aim and primary outcome, or a publicly available protocol. Most formula trials had a high risk of bias for their primary outcome, usually because of inappropriate exclusions of participants from the analysis, and selective reporting. This lack of transparency was complemented by favourable conclusions in more than 90% of recent trials, and evidence of publication bias in recent superiority trials where a favourable standardised mean difference for 84% of primary outcomes was found. The favourable outcomes in equivalence and non-inferiority trials were also affected by selective reporting; almost half of the recent trials designed to show adequate weight gain based on regulatory guidance failed to show equivalence, but authors’ own interpretation of the data supported adequate weight gain in 95% of trials. In most recent formula trials, investigators were employed by, or had financial links to, the formula industry, who were often involved in the statistical analysis and writing. Conflicts of interest could explain some of the transparency issues identified in formula trials because work in other disciplines has shown a relation between commercial funding for clinical trials and favourable results.24
Low rates of breastfeeding are an important global public health issue, formula marketing can adversely affect breastfeeding rates, and the International Code of Marketing of Breast-milk Substitutes is a key international agreement used by policy makers to protect breastfeeding by limiting the marketing of formula.25 In recent formula trials, we found evidence that trial procedures might contravene the code, for example by providing free formula to parents of breastfed or mixed fed infants. Only 12% of trials reported adequate measures to ensure breastfeeding was not undermined in participants of the trial. The code states that “health workers should not give samples of infant formula to pregnant women, mothers of infants and young children, or members of their families”. Therefore, oversight of formula trials might be inadequate, and there might be a failure to adhere to the Declaration of Helsinki by protecting research participants.8 12 26 In a setting where trialists and regulators might require high consumption of formula to better answer scientific uncertainties, there is a risk of creating an incentive structure within a trial that promotes the use of formula in place of breast milk, with substantial health risks for participants in the trial.
Policy implications
Since the invention of industrially produced formula milk in the 19th century, ongoing scientific studies have been conducted aimed at improving formula products, for example by adding vitamins and modifying levels of protein. Clinical trials are potentially important in the development of improved formula products with reduced risks for health and nutrition. In this study, we found the trials to be unreliable, suggesting that much of the recent information generated about formula products might be misleading.3 Given the lack of transparency about the aims of the trials and almost universally favourable conclusions, some trials might have a marketing aim and no robust scientific aim.27 Marketing of formula is cited by the World Health Organization as a major barrier to promoting and protecting breastfeeding. Claims arising from formula trials can contribute to formula marketing by narrowing the perceived benefits of breast milk over formula for consumers. Some have called for a change in the regulatory environment for formula products, including a ban on health and nutrition claims.3 28 Our findings suggest that such changes should include improved oversight, conduct, and reporting of formula trials to ensure they provide a rigorous evidence base to inform nutrition in infants and young children.
Limitations
Our study was a comprehensive evaluation of the conduct and reporting of formula trials. Because of lack of resources, our detailed evaluation was largely restricted to the analysis of the primary outcome for trials published in the peer reviewed literature since 2015. We could not fully analyse all trials completed during this time because many formula trials were unpublished or unregistered, trial authors and funders were reluctant to share protocols, and regulators could not share clinical study reports (personal communication, Andrea Lotze). Our report therefore represents a subset of the formula trials conducted, and the results of many formula trials are not in the public domain. Some groups have found that the Cochrane ROB2 tool is difficult to use and have criticised its reproducibility.29 We found that ROB2 had good reproducibility within our team after initial training and piloting, with κ scores of 0.61-0.87 for blinded assessments of ROB2 domains.
Our assessments of support for breastfeeding were made with criteria developed by an international group of experts in formula trials and their regulation.12 Analysis of pooled standardised mean difference from trials testing varied interventions in diverse populations for heterogeneous outcomes must be interpreted with caution. We are confident that findings were, in general, reported as favourable; but the relations identified between characteristics of the study and standardised mean difference should be regarded as exploratory because of the heterogeneous nature of the data contributing to the analysis of standardised mean difference and small sample sizes of some subgroups.
We did not include formal patient and public involvement in the conduct or reporting of this project but the Delphi consensus guidance which informed the design of this project was undertaken with the involvement of patients and the public.12 We did not formally compare the risk of bias in this area with clinical trials in other areas, but the effect sizes seen in this analysis suggest that the outcomes are at least as favourable as trials of pharmaceuticals licensed for treating general medical or psychiatric conditions.30 Our analyses of primary outcomes from published clinical trials are likely to represent conservative estimates of effect size (standardised mean difference) and selective reporting bias, compared with the outcome analyses selected by authors, because interpretation of primary outcomes by authors was more favourable than our own analysis of the raw data. Our analyses of primary outcomes are also likely to underestimate reporting biases compared with exploratory outcomes or post hoc subgroup analyses, which are often used by the formula industry as the basis of health and nutrition claims.3 31
Conclusion
Our study suggests that formula trials are not reliable and might not adequately protect participants in the trials. The formula industry is closely involved in formula trials, findings are almost always reported as favourable, and little transparency exists about the aims of the trial or reporting of results. Our findings support the need for a substantial change in the conduct and reporting of formula trials to adequately protect participants from harm and protect consumers from misleading information.
What is already known on this topic
Formula milk is consumed by most European and North American infants, and new formula products need to be tested in clinical trials
Concerns have been raised that formula milk trials are biased and could undermine breastfeeding
What this study adds
Most recent formula trials have no prospective record of clearly defined aims of the study
The findings show evidence of publication bias, with more registered unpublished trials than prospectively registered published trials
Most formula trials have a high risk of bias, authors almost always report favourable conclusions, transparency is lacking, and findings are selectively reported
Acknowledgments
We are grateful to Ewa Szymlek-Gay, Huiwen Zhang, Dirk Haller, and Johannes van Goudoever for sharing prospective protocols for their trials, to Manjiang Yao from Nestlé Clinical Research for sharing four retrospective trial protocols and one prospective trial protocol, to Hania Szajewska and to Barbara Marriage from Abbott Laboratories for sharing retrospectively dated protocols, to Mark Corkins and Vanessa Scheeffer for sharing statistical analysis plans, and to Antonia Monayo, Marc Rhoads, Rossella Turco, Tetsuhiro Sakihara, Flavia Indrio, Olle Hernell, Roberto Canani, and Hanna Petersen for sharing undated trial protocols. A summary of an earlier version of information reported in this manuscript was presented by BH at the Star Child Health Forum on 4 February 2021 https://www.starchildhealth.org/past-events.
Web extra.
Extra material supplied by authors
Web appendix 1: Supplementary material
Web appendix 2: Data supplement
Contributors: RJB, JL-B, DI, and BH designed the study, with advice and input from LB, VGL, CMK, and ZD. BH, RJB, AM, MK, FD, JJ, and CP developed and piloted the study search, data extraction, and coding tools. BH conducted the search. BH, RJB, AM, MK, FD, and JJ screened titles, extracted data, and coded outcomes. BH, JL-B, and DI undertook the statistical analyses. RJB wrote the first draft of the manuscript. All authors critically reviewed and revised the manuscript and approved the final manuscript. RJB acts as guarantor of the manuscript. AM and CP made equal contributions to the study. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: This work was funded by Imperial Health Charity, grant FA1920_05. Imperial Health Charity had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report or decision to submit the article for publication. BH and RJB had full access to all of the study data. Final responsibility for the decision to submit for publication was made by RJB and the funder was not involved.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare support from Imperial Health Charity: RJB received personal fees from Cochrane, DBV Technologies, and Prota Therapeutics, and from expert witness work in cases of food anaphylaxis and class actions related to infant formula health claims, outside the submitted work, and received personal fees from Public Health England as a member of the UK Nutrition and Health Claims Committee and the Maternal and Child Nutrition Subgroup of the Scientific Advisory Committee on Nutrition. JL-B received fees from Danone Nutricia Research and the Food Standards Agency, outside of the submitted work. The University of Colorado receives remuneration for LB’s work as senior editor, Cochrane, which is outside the submitted work. The authors report no other relationships or activities that could appear to have influenced the submitted work.
RJB affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned and registered have been explained.
Dissemination to participants and related patient and public communities: The findings of this study will be disseminated to regulators, industry clinical trials departments, and charities which support and advise parents and other carers of young children in making feeding decisions.
Provenance and peer review: Not commissioned, externally peer reviewed.
Ethics statements
Ethical approval
Ethics approval and informed consent were not required for this study. All information used for the analyses was in the public domain, with the exception of the clinical trial protocols and statistical analysis plans which were kindly provided by some trialists and trial sponsors at our request, to confirm methods used in the relevant trials.
Data availability statement
All data collected for this study, including detailed responses to ROB2, breastfeeding, and conflict of interest signalling questions, are in the data supplement (web appendix 2). All other data are in the public domain or publication databases, except for trial protocols and statistical analysis plans received from authors and funders. We have not been given permission to share trial protocols and statistical analysis plans outside of our study team, but they can be requested directly from the trial authors and funders, as indicated in supplementary table 5.
References
- 1. Baker P, Santos T, Neves PA, et al. First-food systems transformations and the ultra-processing of infant and young child diets: The determinants, dynamics and consequences of the global rise in commercial milk formula consumption. Matern Child Nutr 2021;17:e13097. 10.1111/mcn.13097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Tran HT, Nguyen TT, Mathisen R. The use of human donor milk. BMJ 2020;371:m4243. 10.1136/bmj.m4243. [DOI] [Google Scholar]
- 3. Munblit D, Crawley H, Hyde R, Boyle RJ. Health and nutrition claims for infant formula are poorly substantiated and potentially harmful. BMJ 2020;369:m875. 10.1136/bmj.m875. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Infant formula requirements pertaining to current good manufacturing practice, quality control procedures, quality factors, records and reports, and notifications. Electronic Code of Federal Regulations, Title 21, Chapter IB, Part 106.
- 5. Hughes HK, Landa MM, Sharfstein JM. Marketing claims for infant formula: the need for evidence. JAMA Pediatr 2017;171:105-6. 10.1001/jamapediatrics.2016.3837. [DOI] [PubMed] [Google Scholar]
- 6.Tijhuis MJ, Doets EL, Vonk Noordegraaf-Schouten M. Extensive literature search and review as preparatory work for the evaluation of the essential composition of infant and follow-on formulae and growing-up milk. EFSA supporting publications 2014;11:551E.
- 7. World Health Organization . International Code of Marketing of Breast-milk Substitutes. WHO, 1981. [Google Scholar]
- 8. Boyle RJ, Ierodiakonou D, Khan T, et al. Hydrolysed formula and risk of allergic or autoimmune disease: systematic review and meta-analysis. BMJ 2016;352:i974. 10.1136/bmj.i974. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;6:e1000097. 10.1371/journal.pmed.1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Abbasi K. Compulsory registration of clinical trials. BMJ 2004;329:637-8. 10.1136/bmj.329.7467.637. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898. 10.1136/bmj.l4898. [DOI] [PubMed] [Google Scholar]
- 12. Jarrold K, Helfer B, Eskander M, et al. Guidance for the Conduct and Reporting of Clinical Trials of Breast Milk Substitutes. JAMA Pediatr 2020;174:874-81. 10.1001/jamapediatrics.2020.0578. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Bero L, Oostvogel F, Bacchetti P, Lee K. Factors associated with findings of published trials of drug-drug comparisons: why some statins appear more efficacious than others. PLoS Med 2007;4:e184. 10.1371/journal.pmed.0040184. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Chandler J, Cumpston M, Li T, Page MJ, Welch V. Cochrane Handbook for Systematic Reviews of Interventions Version 6.1 2020.
- 15. Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol 2014;14:135. 10.1186/1471-2288-14-135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Duval S, Tweedie R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000;56:455-63. 10.1111/j.0006-341X.2000.00455.x. [DOI] [PubMed] [Google Scholar]
- 17.Leonardi-Bee J, Garcia-Larsen V, Jarrold K, et al. Risk of bias and study conduct in clinical trials of breastmilk substitutes: a systematic review. PROSPERO 2018;CRD42018091928.
- 18. American Academy of Pediatrics Task Force . Clinical testing of infant formulas with respect to nutritional suitability for term infants. 1989. [Google Scholar]
- 19. Fatheree NY, Liu Y, Ferris M, et al. Hypoallergenic formula with Lactobacillus rhamnosus GG for babies with colic: A pilot study of recruitment, retention, and fecal biomarkers. World J Gastrointest Pathophysiol 2016;7:160-70. 10.4291/wjgp.v7.i1.160. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Hulshof L, Overbeek SA, Wyllie AL, et al. Clinical Study Group . Exploring immune development in infants with moderate to severe atopic dermatitis. Front Immunol 2018;9:630. 10.3389/fimmu.2018.00630. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Sakihara T, Otsuji K, Arakaki Y, et al. Randomized trial of early infant formula introduction to prevent cow's milk allergy. J Allergy Clin Immunol 2021;147:224-32. 10.1016/j.jaci.2020.08.021 [DOI] [PubMed] [Google Scholar]
- 22. Szymlek-Gay EA, Domellöf M, Hernell O, et al. Mode of oral iron administration and the amount of iron habitually consumed do not affect iron absorption, systemic iron utilisation or zinc absorption in iron-sufficient infants: a randomised trial. Br J Nutr 2016;116:1046-60. 10.1017/S0007114516003032. [DOI] [PubMed] [Google Scholar]
- 23. Wall CR, Hill RJ, Lovell AL, et al. A multicenter, double-blind, randomized, placebo-controlled trial to evaluate the effect of consuming Growing Up Milk “Lite” on body composition in children aged 12-23 mo. Am J Clin Nutr 2019;109:576-85. 10.1093/ajcn/nqy302. [DOI] [PubMed] [Google Scholar]
- 24. Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev 2017;2:MR000033. 10.1002/14651858.MR000033.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Victora CG, Bahl R, Barros AJ, et al. Lancet Breastfeeding Series Group . Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect. Lancet 2016;387:475-90. 10.1016/S0140-6736(15)01024-7. [DOI] [PubMed] [Google Scholar]
- 26. World Medical Association . Declaration of Helsinki - ethical principles for medical research involving human subjects. WMA, 1964. [Google Scholar]
- 27. Krumholz SD, Egilman DS, Ross JS. Study of neurontin: titrate to effect, profile of safety (STEPS) trial: a narrative account of a gabapentin seeding trial. Arch Intern Med 2011;171:1100-7. 10.1001/archinternmed.2011.241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Hastings G, Angus K, Eadie D, Hunt K. Selling second best: how infant formula marketing works. Global Health 2020;16:77. 10.1186/s12992-020-00597-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Minozzi S, Cinquini M, Gianola S, Gonzalez-Lorenzo M, Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol 2020;126:37-44. 10.1016/j.jclinepi.2020.06.015. [DOI] [PubMed] [Google Scholar]
- 30. Leucht S, Hierl S, Kissling W, Dold M, Davis JM. Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses. Br J Psychiatry 2012;200:97-106. 10.1192/bjp.bp.111.096594. [DOI] [PubMed] [Google Scholar]
- 31. Boyle R, Brown N, Chiang WC, et al. Retraction Note to: Partially hydrolysed, prebiotic supplemented whey formula for the prevention of allergic manifestations in high risk infants: a multicentre double-blind randomised controlled trial. Clin Transl Allergy 2020;10:48. 10.1186/s13601-020-00356-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Web appendix 1: Supplementary material
Web appendix 2: Data supplement
Data Availability Statement
All data collected for this study, including detailed responses to ROB2, breastfeeding, and conflict of interest signalling questions, are in the data supplement (web appendix 2). All other data are in the public domain or publication databases, except for trial protocols and statistical analysis plans received from authors and funders. We have not been given permission to share trial protocols and statistical analysis plans outside of our study team, but they can be requested directly from the trial authors and funders, as indicated in supplementary table 5.