An Empirical Evaluation of the Impact Scenario of Pooling Bodies of Evidence from Randomized Controlled Trials and Cohort Studies in Nutrition Research

ABSTRACT

Only very few Cochrane nutrition reviews include cohort studies (CSs), but most evidence in nutrition research comes from CSs. We aimed to pool bodies of evidence (BoE) from randomized controlled trials (RCTs) derived from Cochrane reviews with matched BoE from CSs. The Cochrane Database of Systematic Reviews and MEDLINE were searched for systematic reviews (SRs) of RCTs and SRs of CSs. BoE from RCTs were pooled together with BoE from CSs using random-effects and common-effect models. Heterogeneity, 95% prediction intervals, contributed weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from CSs modified the conclusion from BoE of RCTs were evaluated. Overall, 80 diet–disease outcome pairs based on 773 RCTs and 720 CSs were pooled. By pooling BoE from RCTs and CSs with a random-effects model, for 45 (56%) out of 80 diet–disease associations the 95% CI excluded no effect and showed mainly a reduced risk/inverse association. By pooling BoE from RCTs and CSs, median I² = 46% and the median contributed weight of RCTs to the pooled estimates was 34%. The direction of effect between BoE from RCTs and pooled effect estimates was rarely opposite (n = 17; 21%). The integration of BoE from CSs modified the result (by examining the 95% CI) from BoE of RCTs in 35 (44%) of the 80 diet–disease associations.

Our pooling scenario showed that the integration of BoE from CSs modified the conclusion from BoE of RCTs in nearly 50% of the associations, although the direction of effect was mainly concordant between BoE of RCTs and pooled estimates. Our findings provide insights for the potential impact of pooling both BoE in Cochrane nutrition reviews. CSs should be considered for inclusion in future Cochrane nutrition reviews, and we recommend analyzing RCTs and CSs in separate meta-analyses, or, if combined together, with a subgroup analysis.

Statement of Significance: Our pooling scenario showed that the integration of bodies of evidence from cohort studies modified the conclusion from bodies of evidence of RCTs in nearly 50% of the associations, although the direction of effect was mainly concordant between bodies of evidence of RCTs and pooled estimates.

Introduction

The Global Burden of Disease study group indicated that noncommunicable diseases (NCDs) accounted for 73% of deaths worldwide (1), and evidence from systematic reviews (SRs) of cohort studies (CSs) showed that suboptimal diet accounted for ∼20% of all deaths worldwide (2). CSs that evaluate patient-relevant outcomes (e.g., NCDs) provide important insights into diet–disease relations and, because evidence from RCTs is often not available, commonly inform dietary guidelines for the primary prevention of NCDs (3, 4). Randomized controlled trials (RCTs), if well-designed and well-conducted, give robust answers to the research questions they address and are widely encouraged as the ideal methodology for causal inference (5); however, dietary RCTs also suffer from inherent methodological limitations (4). Such limitations include for example the impossibility of ensuring that participants are unaware of their dietary regimen (except for placebo-controlled RCTs of dietary supplements), or the often observed low adherence to a specific dietary regimen. In contrast to RCTs, large CSs may often have higher external validity, and be able to investigate the long-term association of lifestyle behaviors with patient-relevant outcomes. However, core limitations of CSs include bias due to prevalent-user designs, inappropriate comparators, residual confounding, and measurement error (4).

Nevertheless, it is generally considered that SRs should be based on RCTs because these studies are more likely to provide unbiased information than other study designs. The Cochrane Database of Systematic Reviews is the leading resource for SRs in health care with a clear focus on bodies of evidence (BoE) from RCTs, and internationally recognized as the highest standard in evidence-based health care.

Approximately 10% of all Cochrane reviews are nutrition reviews (6). In a cross-sectional study it was shown that only very few Cochrane nutrition reviews (2%) include observational studies (6), likely because Cochrane reviews focus on research questions related to causal effect and effectiveness, where RCTs are considered the “gold standard.” However, this has been criticized in the past and is motivated by the principle of using the best available evidence, which might stem from observational studies if RCTs are missing or scarce (7). Because most evidence in nutrition research comes from CSs, BoE from CSs can complement BoE from RCTs, and vice versa. However, the potential impact of integrating BoE of CSs in Cochrane nutrition evidence syntheses has not been investigated yet.

To close this important research gap, we aimed to conduct a pooling scenario of BoE from RCTs derived from Cochrane reviews with matched BoE from CSs in this empirical study. In order to shed light on the potential impact of integrating BoE from CSs into the effect estimates derived from BoE of RCTs, we will investigate to what extent the integration of BoE from CSs modified the conclusion from BoE of RCTs, its direction of effect, and its impact on statistical inconsistency. Moreover, we will also evaluate the contributed aggregated weights of RCTs to the pooled estimates, use a random-effects and a common-effect model for pooling, calculate 95% prediction intervals (PIs), and test for subgroup differences between BoE from RCTs and CSs.

Methods

This study was planned, written, and reported in adherence to guidelines for reporting meta-epidemiologic methodology research (8). Table 1 describes the inclusion criteria [patients/population, intervention/exposure, comparator, and outcome (PI/ECO)].

TABLE 1.

Detailed description of inclusion criteria¹

Population	Generally healthy participants (children, adolescents, and adults)
Intervention/exposure	Dietary pattern: e.g., Mediterranean diet, Dietary Approaches to Stop Hypertension, low-carbohydrate diet. Food groups: the following food groups (macro-level) and foods (micro-level), e.g., grains, vegetables, fruit, milk and dairy products, meat, processed meat, fish, eggs, nuts, chocolate, oils, were considered. Macronutrients: carbohydrate (starch, fructose, glucose, sucrose); fat: e.g., ω-3 fatty acids (EPA, DHA, α-linolenic acid), ω-6 fatty acids (linoleic acid), monounsaturated fat; protein (e.g., amino acids). Micronutrients: vitamins: β-carotene; vitamins A, E, C (ascorbic acid), and D (cholecalciferol, ergocalciferol); B vitamins (thiamin, riboflavin, niacin, pyridoxine, cobalamin, folic acid). Minerals: magnesium, calcium, selenium, sodium, potassium, iron, zinc, copper, iodine. Other: fiber (psyllium, inulin, cellulose); probiotics; prebiotics; and synbiotics.
Control/comparison	Low (no) intake (status) level of the foregoing interventions/exposure. Placebo/usual care.
Outcomes	e.g. all-cause mortality, cardiovascular disease, ischemic heart disease (myocardial infarction, ischemic heart disease, and acute coronary syndrome), stroke, cancer, type 2 diabetes, dementia, fractures, age-related macular degeneration, anthropometric outcomes; important intermediate disease markers such as systolic blood pressure, diastolic blood pressure, fasting glucose, and LDL cholesterol.
Study design	Systematic reviews of randomized controlled trials. Matching systematic reviews of CSs: CSs (if available, prospective CSs were preferred).

¹CS, cohort study.

Identification of SRs of RCTs

We searched for SRs of RCTs in the Cochrane Database of Systematic Reviews published between 1 January, 2010 and 31 December, 2019 (Supplemental Appendix A). Screening of titles/abstracts was done by 1 reviewer (LS), and was followed by a screening for inclusion of relevant full articles by 2 reviewers independently (LS and JZ). Discrepancies were resolved by an additional reviewer (JJM).

Identification of matching SRs of CSs

After all potentially relevant SRs of RCTs were identified we searched for matching SRs of CSs as counterpart. First, we screened whether eligible Cochrane reviews included CSs. Second, we conducted searches for SRs of CSs in MEDLINE, published within the last 10 y (Supplemental Appendix B). We selected a time period of 10 y to ensure comparability between the 2 BoE. Screening of titles/abstracts was conducted by 1 reviewer (LS), and was followed by a screening for inclusion of relevant full articles by 2 reviewers independently (LS and JZ). By hand searching additional matching SRs of CSs were identified. The most appropriate (investigating similar PI/ECO) and comprehensive (most recent) matching SRs of CSs were selected.

Matching SRs of RCTs with SRs of CSs according to PI/ECO criteria

For all potentially eligible SRs of CSs 2 reviewers judged whether each PI/ECO-element matched those of the corresponding SRs of RCTs as “more or less identical” (very closely matched), “similar but not identical” (closely matched), or “broadly similar” (matched, but less close) (9). Based on these criteria we classified each eligible effect estimate within an SR of CSs relative to its effect estimate within an SR of RCTs as (overall rating) “more or less identical,” “similar but not identical,” and “broadly similar.” For each eligible SR of RCTs we matched a maximum of 6 outcomes (max. 3 patient-relevant outcomes; and max. 3 intermediate disease outcomes) for a given intervention/exposure. Selection of outcomes was based on the ranking in the summary of findings tables in the identified Cochrane reviews (from top to bottom). Supplemental Tables 1 and 2 report the matching classifications, and a detailed description of the matching process can be found elsewhere (10).

Data extraction

We extracted the following data for each included outcome pair (e.g., all-cause mortality, cardiovascular disease, stroke, type 2 diabetes) of a BoE from RCTs and matched CSs: name of first author, year of publication, type of intervention/exposure (dietary pattern, food group/food, macronutrient, micronutrient, other), description of comparator (placebo, lowest intake/status category, control diet), adjusted (when available) effect estimates [risk ratio (RR), HR, OR, mean difference (MD), 95% CI], type of comparison (e.g., high compared with low, dose-response), and number of studies included. A detailed description of the data extraction can be found elsewhere (10, 11). For the current analysis all effect estimates and corresponding 95% CIs of the primary studies included for a relevant BoE were extracted. Primary studies based on inappropriate study designs (i.e., case-control, cross-sectional studies, retrospective CSs, and quasi-RCTs) were excluded.

Statistical analysis

For the current analysis we pooled first the relevant primary studies of each eligible BoE derived from RCTs with a random-effects model. Second, we pooled the relevant primary studies of a matched BoE derived from CSs with a random-effects model. Third, we pooled the BoE from RCTs with the BoE from CSs with a random-effects model (a common-effect model was used as a sensitivity analysis) for each identified matched diet–disease association (Supplemental Figures 1–80). For the analysis, binary outcomes (pooled as RRs, HRs, and ORs) and continuous outcomes [pooled as MDs on the same scale, e.g., blood pressure (mm Hg) or body weight (kg) was used in a meta-analysis (MA)] were considered.

When individual effect sizes were correlated, we used the equations recommended by Borenstein et al. (12) to convert correlated outcomes. Overall, we identified 3 MAs of cohort studies (13, 14) which included primary studies with correlated outcomes, and we converted the corresponding effect sizes (Supplemental Figures 2, 8, and 36).

Random-effects models were used for all MAs to account for potential between-study heterogeneity. We explored the impact of including CSs on pooled effect estimates by combining BoE from RCTs and CSs (with or without subgroups). To do so, we compared the results and conclusions (examining 95% CIs including compared with excluding no effect) between the BoE of RCTs only and that including both RCTs and CSs. Finally, we evaluated the contributed weight of RCTs to the pooled estimates, and conducted a test for subgroup differences (statistical significance: P < 0.05 for subgroup test) between the 2 types of BoE.

Heterogeneity in MAs was tested with a standard χ² test. The I² parameter was used to quantify any inconsistency: I² =100% × (Q - df) / Q, where Q is the χ² statistic and df is its degrees of freedom (15). An I² value >50% was considered to represent considerable heterogeneity (16). However, because I² is dependent on the study size (it increases with increasing study size), we also calculated τ² for binary outcomes, which is independent of study size and describes variability between studies in relation to the risk estimates (17). We did not calculate τ² for continuous outcomes owing to the use of different scales between MAs. MAs were conducted using Review Manager (RevMan) version 5.3 (18).

For the summary random effects we estimated for each MA also the 95% PI, which further accounts for the degree of between-study heterogeneity and gives a range for which we are 95% confident that the effect in a new study examining the same association lies within it (17). 95% PI calculations were conducted with Stata 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC).

Results

Overall, 33 SRs of RCTs (Cochrane reviews) (19–51) and 46 matching SRs of CSs were included (13, 14, 52–95). Two Cochrane reviews included also CSs (19, 20). Of the identified 97 diet–disease outcome pairs (Supplemental Tables 2 and 3), 80 (overall 160 effect estimates were recalculated) were included in the present pooling scenario (68 dichotomous and 12 continuous) (Supplemental Figures 1–80). Seventeen outcome pairs were excluded from the current analysis (Supplemental Table 4 provides reasons for exclusion).

The 160 considered effect estimates were based on 773 RCTs and 720 CSs. Detailed study characteristics including description of population, age, description of intervention/comparator, outcomes, range of study length, and risk of bias (RoB)/study quality of the primary studies included in each diet–disease association have been described in detail elsewhere (10, 11).

Fifty-six of the diet–disease associations were classified (PI/ECO similarity degree) as “similar but not identical,” whereas 24 were classified as “broadly similar” (Table 2). Out of the 80 BoE from RCTs, for 17 (21%) the 95% CI excluded no effect (16 showed a risk-reducing effect/-lowering effect), whereas out of the 80 BoE from CSs, 43 (54%) indicated a 95% CI excluding no effect. Seven (9%) out of 80 diet–disease associations showed for both BoE a 95% CI excluding no effect, and the associations were in the same direction. The median I² was 0% (τ² = 0) across BoE from RCTs and 55% (τ² = 0.01) across BoE from CSs, whereas the mean I² was 20% (τ² = 0.02) and 47% (τ² = 0.02), respectively. Table 2, Figure 1 (all-cause mortality), and Figure 2 (cardiovascular disease) show the summary effects of the BoE from RCTs, CSs, and the pooling scenario.

TABLE 2.

Overview of the effect estimates of 80 included diet–disease outcome pairs, including pooling results of BoE from RCTs and CSs based on RE and CE models, 95% PI, heterogeneity, test for subgroup difference, and PI/ECO similarity degree¹

Authors (reference), BoE RCTs	Authors (reference), BoE CSs	Intervention/exposure category	Outcome category	BoE RCTs, n	Effect estimate (95% CI)	I 2 (%)/τ	BoE CSs, n	Effect estimate (95% CI)	I 2 (%)/τ	Pooled effect estimate (95% CI) RE (95% PI)	I 2 (%)/τ	Weight RCTs, %	RCT conclusion modified by pooling (Yes/No)	Test for subgroup difference (P value)	Pooled effect estimate (95% CI) CE	Degree of PI/ECO similarity+
Abdelhamid et al. (21)	Chowdhury et al. (53)	ω-3 fatty acids	Cardiovascular disease	38	RR: 0.99 (0.94, 1.04)	37/0.00	16	RR: 0.87 (0.78, 0.97)	76/0.03	RR: 0.95 (0.90, 0.99) (0.76, 1.18)	58/0.01	58.8	Y	0.03	RR: 0.96 (0.94, 0.99)	2
Abdelhamid et al. (21)	Pan et al. (13)	α-Linolenic acid	Cardiovascular disease	5	RR: 0.95 (0.83, 1.07)	0/0.00	11	RR: 0.93 (0.85, 1.03)	41/0.01	RR: 0.94 (0.87, 1.02) (0.77, 1.15)	30/0.01	20.6	N	0.89	RR: 0.96 (0.91, 1.02)	2
Abdelhamid et al. (21)	Wan et al. (52)	ω-3 fatty acids	All-cause mortality	39	RR: 0.98 (0.93, 1.03)	11/0.00	6	RR: 0.86 (0.80, 0.93)	56/0.00	RR: 0.93 (0.88, 0.98) (0.79, 1.10)	34/0.01	59.7	Y	0.008	RR: 0.93 (0.90, 0.95)	2
Abdelhamid et al. (21)	Wei et al. (56)	α-Linolenic acid	Cardiovascular mortality	4	RR: 0.96 (0.74, 1.25)	0/0.00	9	RR: 0.85 (0.75, 0.96)	16/0.01	RR: 0.85 (0.78, 0.93) (0.77, 0.94)	0/0.00	11.2	Y	0.40	RR: 0.85 (0.78, 0.93)	2
Abdelhamid et al. (21)	Wei et al. (56)	α-Linolenic acid	Ischemic heart disease	4	RR: 1.00 (0.82, 1.22)	2/0.00	13	RR: 0.91 (0.85, 0.97)	6/0.00	RR: 0.91 (0.86, 0.97) (0.84, 1.00)	4/0.00	9.8	Y	0.37	RR: 0.91 (0.86, 0.97)	2
Abdelhamid et al. (22)	Li et al. (55)	Polyunsaturated fat	All-cause mortality	24	RR: 0.98 (0.89, 1.07)	0/0.00	11	RR: 0.87 (0.81, 0.94)	68/0.01	RR: 0.90 (0.85, 0.96) (0.77, 1.06)	38/0.01	29.4	Y	0.06	RR: 0.88 (0.86, 0.90)	2
Abdelhamid et al. (22)	Chowdhury et al. (53)	Polyunsaturated fat	Ischemic heart disease	15	RR: 0.87 (0.72, 1.06)	45/0.04	8	RR: 0.98 (0.90, 1.07)	54/0.01	RR: 0.95 (0.87, 1.03) (0.75, 1.20)	49/0.01	33.4	N	0.30	RR: 0.97 (0.93, 1.00)	2
Abdelhamid et al. (22)	Zhu et al. (14)	Polyunsaturated fat	Cardiovascular disease	2	RR: 0.84 (0.60, 1.20)	79/0.05	30	RR: 0.96 (0.92, 1.00)	59/0.00	RR: 0.96 (0.92, 1.00) (0.84, 1.09)	60/0.00	6.2	N	0.47	RR: 1.00 (0.99, 1.01)	2
Adler et al. (23)	Aburto et al. (57)	Low sodium	All-cause mortality	7	RR: 0.96 (0.84, 1.11)	0/0.00	2	RR: 0.95 (0.71, 1.27)	82/0.05	RR: 0.94 (0.81, 1.08) (0.68, 1.30)	38/0.01	39.6	N	0.94	RR: 0.94 (0.86, 1.02)	2
Adler et al. (23)	Aburto et al. (57)	Low sodium	Cardiovascular mortality	3	RR: 0.67 (0.45, 1.01)	0/0.00	3	RR: 0.87 (0.64, 1.18)	79/0.07	RR: 0.82 (0.65, 1.04) (0.43, 1.58)	61/0.05	19.3	N	0.33	RR: 0.77 (0.69, 0.87)	2
Adler et al. (23)	Aburto et al. (57)	Low sodium	Cardiovascular disease	4	RR: 0.76 (0.57, 1.02)	0/0.00	3	RR: 0.87 (0.64, 1.18)	79/0.07	RR: 0.83 (0.67, 1.03) (0.46, 1.30)	59/0.05	30.8	N	0.55	RR: 0.78 (0.69, 0.87)	2
Adler et al. (23)	Leyvraz et al. (58)	Low sodium	Systolic blood pressure (mm Hg)	6	MD: −1.79 (−3.23, −0.36)	74	1	MD: −1.20 (−1.50, −0.90)	NA	MD: −1.59 (−2.49, −0.69) (−4.27, 1.08)	70	71.7	N	0.43	MD: −1.28 (−1.55, −1.00)	3
Adler et al. (23)	Leyvraz et al. (58)	Low sodium	Diastolic blood pressure (mm Hg)	5	MD: −1.17 (−2.08, −0.26)	58	1	MD: 1.20 (0.95, 1.45)	NA	MD: −0.82 (−2.27, 0.63) (−5.73, 4.08)	93	78.9	Y	<0.0001	MD: 0.76 (0.54, 0.98)	3
Al-Khudairy et al. (24)	Aune et al. (59)	Vitamin C	Cardiovascular disease	1	HR: 0.99 (0.89, 1.10)	NA	9	RR: 0.84 (0.78, 0.91)	0/0.00	HR/RR: 0.88 (0.80, 0.96) (0.72, 1.07)	29/0.01	25.1	Y	0.02	HR/RR: 0.89 (0.84, 0.95)	2
Al-Khudairy et al. (24)	Aune et al. (59)	Vitamin C	All-cause mortality	1	HR: 1.07 (0.97, 1.18)	NA	16	RR: 0.86 (0.80, 0.92)	69/0.01	HR/RR: 0.88 (0.82, 0.94) (0.70, 1.10)	71/0.01	9.3	Y	0.0004	HR/RR: 0.95 (0.92, 0.97)	2
Avenell et al. (25)	Feng et al. (60)	Vitamin D	Hip fracture	10	RR: 1.12 (0.97, 1.30)	0/0.00	11	RR: 0.62 (0.53, 0.71)	17/0.01	RR: 0.78 (0.65, 0.93) (0.41, 1.48)	62/0.09	41.8	Y	<0.00001	RR: 0.80 (0.73, 0.88)	3
Avenell et al. (25)	Feng et al. (60)	Vitamin D	Any fracture	14	RR: 1.04 (0.95, 1.15)	18/0.01	11	RR: 0.71 (0.58, 0.86)	72/0.06	RR: 0.89 (0.80, 0.99) (0.61, 1.30)	60/0.03	54.5	Y	0.0005	RR: 0.96 (0.91, 1.01)	3
Bjelakovic et al. (26)	Aune et al. (59)	β-Carotene	All-cause mortality	31	RR: 1.02 (0.98, 1.07)	34/0.00	8	RR: 0.82 (0.78, 0.87)	0/0.00	RR: 0.95 (0.90, 1.01) (0.76, 1.19)	67/0.01	73.0	N	<0.00001	RR: 1.02 (0.99, 1.04)	2
Bjelakovic et al. (26)	Aune et al. (59)	Vitamin E	All-cause mortality	64	RR: 1.02 (0.99, 1.04)	0/0.00	9	RR: 0.98 (0.92, 1.04)	6/0.00	RR: 1.01 (0.99, 1.03) (0.99, 1.03)	0/0.00	83.4	N	0.27	RR: 1.01 (0.99, 1.03)	2
Bjelakovic et al. (26)	Aune et al. (59)	Vitamin C	All-cause mortality	41	RR: 1.01 (0.97, 1.05)	0/0.00	16	RR: 0.86 (0.80, 0.92)	69/0.01	RR: 0.92 (0.88, 0.96) (0.78, 1.08)	40/0.01	42.6	Y	0.0001	RR: 0.96 (0.93, 0.98)	2
Bjelakovic et al. (26)	Aune et al. (59)	Vitamin A	All-cause mortality	18	RR: 1.04 (0.96, 1.13)	25/0.00	8	RR: 0.82 (0.78, 0.87)	0/0.00	RR: 0.93 (0.85, 1.02) (0.66, 1.30)	71/0.02	52.6	N	<0.00001	RR: 0.97 (0.94, 1.01)	3
Bjelakovic et al. (28)	Chowdhury et al. (61)	Vitamin D	All-cause mortality	56	RR: 0.97 (0.94, 1.00)	0/0.00	68	RR: 0.70 (0.65, 0.75)	83/0.05	RR: 0.76 (0.72, 0.81) (0.48, 1.22)	84/0.05	28.0	Y	<0.00001	RR: 0.79 (0.77, 0.80)	3
Bjelakovic et al. (28)	Chowdhury et al. (61)	Vitamin D	Cardiovascular mortality	10	RR: 0.98 (0.90, 1.07)	0/0.00	29	RR: 0.69 (0.60, 0.79)	84/0.10	RR: 0.73 (0.65, 0.83) (0.39, 1.38)	83/0.09	14.6	Y	<0.0001	RR: 0.75 (0.72, 0.79)	3
Bjelakovic et al. (28)	Han et al. (62)	Vitamin D	Cancer mortality	4	RR: 0.88 (0.78, 0.98)	0/0.00	16	RR: 0.81 (0.71, 0.93)	49/0.04	RR: 0.83 (0.75, 0.92) (0.61, 1.12)	41/0.02	27.2	N	0.39	RR: 0.83 (0.78, 0.88)	3
Bjelakovic et al. (27)	Han et al. (62)	Vitamin D	Cancer	18	RR: 1.00 (0.94, 1.06)	0/0.00	8	RR: 0.86 (0.73, 1.02)	71/0.03	RR: 0.93 (0.85, 1.02) (0.73, 1.19)	38/0.01	44.7	N	0.11	RR: 0.96 (0.91, 1.01)	3
Bjelakovic et al. (27)	Hossain et al. (63)	Vitamin D	Breast cancer	7	RR: 0.97 (0.86, 1.09)	0/0.00	2	RR: 0.94 (0.87, 1.02)	69/0.00	RR: 0.97 (0.95, 0.99) (0.94, 0.99)	0/0.00	3.9	Y	0.68	RR: 0.97 (0.95, 0.99)	2
Bjelakovic et al. (27)	Zhang et al. (64)	Vitamin D	Lung cancer	5	RR: 0.86 (0.69, 1.07)	0/0.00	3	RR: 0.89 (0.77, 1.03)	0/0.00	RR: 0.88 (0.78, 0.99) (0.75, 1.02)	0/0.00	31.4	Y	0.81	RR: 0.88 (0.78, 0.99)	2
De-Regil et al. (51)	Blencowe et al. (93)	Folate	Neural tube defect	5	RR: 0.31 (0.17, 0.58)	0/0.00	3	RR: 0.37 (0.23, 0.58)	30/0.06	RR: 0.39 (0.30, 0.50) (0.28, 0.54)	0/0.00	18.0	N	0.68	RR: 0.39 (0.30, 0.50)	2
De-Regil et al. (51)	Feng et al. (92)	Folate	Congenital cardiovascular anomalies	3	RR: 0.57 (0.24, 1.33)	0/0.00	1	RR: 0.60 (0.38, 0.96)	NA	RR: 0.59 (0.39, 0.89) (0.24, 1.46)	0/0.00	23.1	Y	0.91	RR: 0.59 (0.39, 0.89)	2
Hemmingsen et al. (33)	Schwingshackl et al. (68)	Healthy diet	Type 2 diabetes	1	RR: 0.65 (0.52, 0.81)	NA	10	RR: 0.82 (0.78, 0.85)	72/0.01	RR: 0.81 (0.78, 0.85) (0.68, 0.97)	72/0.01	2.5	N	0.05	RR: 0.81 (0.79, 0.83)	2
Hemmingsen et al. (33)	Schwingshackl et al. (68)	Healthy diet	All-cause mortality	1	RR: 1.02 (0.21, 4.98)	NA	13	RR: 0.78 (0.77, 0.80)	59/0.00	RR: 0.78 (0.77, 0.80) (0.73, 0.84)	58/0.00	0	Y	0.74	RR: 0.78 (0.77, 0.79)	2
Hofmeyr et al. (34)	Newberry et al. (69)	Calcium	Pre-eclampsia	13	RR: 0.47 (0.33, 0.68)	70/0.18	2	RR: 0.97 (0.78, 1.21)	13/0.01	RR: 0.59 (0.45, 0.78) (0.27, 1.31)	69/0.11	73.6	N	0.0008	RR: 0.85 (0.76, 0.95)	2
Hofmeyr et al. (34)	Newberry et al. (69)	Calcium	High blood pressure	12	RR: 0.65 (0.53, 0.81)	74/0.06	2	RR: 1.12 (0.83, 1.50)	66/0.03	RR: 0.76 (0.64, 0.90) (0.45, 1.28)	74/0.05	75.7	N	0.004	RR: 0.91 (0.85, 0.96)	2
Hooper et al. (35)	Noto et al. (71)	Low fat/modified fat	Cardiovascular mortality	14	RR: 0.94 (0.85, 1.04)	0/0.00	3	RR: 0.91 (0.81, 1.03)	0/0.00	RR: 0.93 (0.86, 1.00) (0.85, 1.01)	0/0.00	57.7	N	0.69	RR: 0.93 (0.86, 1.00)	2
Hooper et al. (35)	Seidelmann et al. (70)	Low fat/modified fat	All-cause mortality	20	RR: 0.98 (0.93, 1.04)	0/0.00	6	RR: 0.83 (0.75, 0.92)	40/0.00	RR: 0.92 (0.87, 0.98) (0.80, 1.06)	22/0.00	60.1	Y	0.005	RR: 0.92 (0.88, 0.96)	2
Hooper et al. (35)	Zhu et al. (14)	Low fat/modified fat	Cardiovascular disease	18	RR: 0.86 (0.77, 0.96)	50/0.02	32	RR: 1.03 (0.99, 1.07)	56/0.01	RR: 1.00 (0.96, 1.04) (0.83, 1.20)	56/0.01	21.2	Y	0.002	RR: 1.00 (0.98, 1.01)	2
Hooper et al. (36)	de Souza et al. (73)	Low saturated fat	All-cause mortality	11	RR: 0.97 (0.90, 1.05)	3/0.00	5	RR: 1.01 (0.92, 1.10)	33/0.00	RR: 0.99 (0.94, 1.05) (0.88, 1.12)	18/0.00	44.4	N	0.57	RR: 1.01 (0.96, 1.05)	2
Hooper et al. (36)	de Souza et al. (73)	Low saturated fat	Cardiovascular mortality	10	RR: 0.95 (0.80, 1.12)	30/0.02	3	RR: 1.03 (0.89, 1.18)	18/0.00	RR: 0.99 (0.89, 1.10) (0.77, 1.26)	25/0.01	56.7	N	0.46	RR: 0.98 (0.91, 1.06)	2
Hooper et al. (36)	de Souza et al. (73)	Low saturated fat	Cardiovascular disease	11	RR: 0.83 (0.72, 0.96)	65/0.03	12	RR: 0.95 (0.86, 1.05)	47/0.02	RR: 0.90 (0.83, 0.98) (0.67, 1.23)	55/0.02	39.2	N	0.15	RR: 0.94 (0.90, 0.98)	2
Hooper et al. (38)	Chowdhury et al. (53)	ω-6 fatty acids	Cardiovascular disease	7	RR: 0.97 (0.81, 1.15)	45/0.02	8	RR: 0.98 (0.90, 1.06)	54/0.01	RR: 0.97 (0.91, 1.04) (0.81, 1.17)	46/0.01	29.5	N	0.91	RR: 0.98 (0.94, 1.01)	2
Hooper et al. (38)	Li et al. (55)	ω-6 fatty acids	All-cause mortality	10	RR: 1.00 (0.88, 1.12)	0/0.00	11	RR: 0.87 (0.81, 0.94)	68/0.01	RR: 0.90 (0.84, 0.96) (0.75, 1.08)	54/0.01	20.1	Y	0.07	RR: 0.88 (0.86, 0.90)	2
Hooper et al. (38)	Li et al. (55)	ω-6 fatty acids	Cardiovascular mortality	7	RR: 1.09 (0.76, 1.55)	61/0.1	14	RR: 0.86 (0.81, 0.92)	6/0.00	RR: 0.89 (0.81, 0.98) (0.69, 1.14)	39/0.01	21.1	Y	0.21	RR: 0.89 (0.85, 0.93)	2
Jin et al. (20)	Jin et al. (20)	Flavonoids	Colorectal adenoma/cancer	1	RR: 1.09 (0.93, 1.28)	NA	3	RR: 1.00 (0.80, 1.25)	66/0.02	RR: 1.03 (0.88, 1.20) (0.56, 1.88)	56/0.01	30.4	N	0.55	RR: 1.02 (0.93, 1.13)	3
Jin et al. (20)	Jin et al. (20)	Isoflavones	Colorectal adenoma/cancer	1	RR: 0.98 (0.83, 1.16)	NA	1	RR: 1.16 (0.96, 1.41)	NA	RR: 1.06 (0.90, 1.25) (NA)	43/0.01	54.1	N	0.19	RR: 1.06 (0.93, 1.19)	3
Jin et al. (20)	Jin et al. (20)	Flavonols	Colorectal adenoma/cancer	1	RR: 0.94 (0.80, 1.10)	NA	1	RR: 0.95 (0.83, 1.08)	NA	RR: 0.94 (0.85, 1.04) (NA)	0/0.01	39.5	N	0.93	RR: 0.94 (0.85, 1.04)	3
Keats et al. (50)	Wolf et al. (94)	Micronutrients	Preterm birth	18	RR: 0.95 (0.89, 1.01)	51/0.01	4	RR: 0.84 (0.69, 1.03)	73/0.03	RR: 0.93 (0.88, 0.99) (0.77, 1.12)	58/0.01	80.4	Y	0.26	RR: 0.95 (0.92, 0.97)	2
Keats et al. (50)	Wolf et al. (94)	Micronutrients	Low birth weight	18	RR: 0.88 (0.85, 0.91)	0/0.00	2	RR: 0.79 (0.45, 1.41)	89/0.15	RR: 0.88 (0.84, 0.92) (0.81, 0.95)	9/0.00	94.4	N	0.72	RR: 0.88 (0.85, 0.91)	2
Keats et al. (50)	Wolf et al. (94)	Micronutrients	Small gestational age	17	RR: 0.92 (0.87, 0.97)	40/0.00	3	RR: 0.77 (0.63, 0.93)	43/0.01	RR: 0.89 (0.83, 0.95) (0.70, 1.12)	69/0.01	85.8	N	0.07	RR: 0.96 (0.94, 0.98)	2
Kelly et al. (39)	Ye et al. (74)	Whole grains	Body weight, kg	5	MD: −0.41 (−1.04, 0.23)	0	3	MD: −0.30 (−0.37, −0.24)	99	MD: −0.31 (−0.37, −0.24) (−0.46, −0.15)	98	1.0	Y	0.76	MD: −0.33 (−0.34, −0.33)	2
Mathew et al. (40)	Jiang et al. (75)	β-Carotene	Cataract	2	RR: 0.99 (0.91, 1.08)	0/0.00	7	RR: 0.90 (0.83, 0.99)	0/0.00	RR: 0.95 (0.90, 1.01) (0.88, 1.02)	0/0.00	53.9	N	0.12	RR: 0.95 (0.90, 1.01)	2
Mathew et al. (40)	Jiang et al. (75)	Vitamin E	Cataract	3	RR: 0.97 (0.91, 1.04)	0/0.00	6	RR: 0.88 (0.75, 1.03)	31/0.01	RR: 0.94 (0.88, 1.01) (0.84, 1.06)	12/0.00	67.5	N	0.25	RR: 0.95 (0.90, 1.00)	2
Mathew et al. (40)	Jiang et al. (75)	Vitamin C	Cataract	1	RR: 1.02 (0.91, 1.14)	NA	7	RR: 0.74 (0.59, 0.95)	78/0.07	RR: 0.79 (0.64, 0.97) (0.41, 1.50)	81/0.06	17.5	Y	0.02	RR: 0.88 (0.82, 0.95)	2
Palacios et al. (41)	Hu et al. (77)	Vitamin D	Gestational diabetes	5	RR: 0.54 (0.34, 0.86)	0/0.00	21	OR: 0.76 (0.64, 0.90)	61/0.08	RR/OR: 0.74 (0.63, 0.87) (0.42, 1.31)	54/0.07	7.8	N	0.18	RR/OR: 0.73 (0.67, 0.80)	3
Palacios et al. (41)	Tous et al. (78)	Vitamin D	Preterm birth	4	RR: 1.25 (0.92, 1.69)	0/0.00	19	OR: 0.77 (0.65, 0.92)	63/0.08	RR/OR: 0.82 (0.69, 0.98) (0.43, 1.57)	63/0.09	13.6	Y	0.008	RR/OR: 0.77 (0.70, 0.84)	3
Palacios et al. (41)	Tous et al. (78)	Vitamin D	Birth length, cm	11	MD: −0.04 (−0.26, 0.19)	23	7	MD: −0.12 (−0.33, 0.09)	62	MD: −0.08 (−0.23, 0.07) (−0.50, 0.34)	41	41.1	N	0.60	MD: −0.06 (−0.16, 0.03)	3
Palacios et al. (41)	Tous et al. (78)	Vitamin D	Birth weight, g	13	MD: 32.61 (−9.51, 74.72)	22	14	MD: 84.20 (52.59, 115.81)	58	MD: 68.33 (40.42, 96.24) (−34.39, 171.05)	55	33.6	Y	0.05	MD: 73.53 (57.69, 89.37)	3
Palacios et al. (41)	Tous et al. (78)	Vitamin D	Head circumference at birth, cm	10	MD: 0.08 (−0.09, 0.25)	40	7	MD: 0.47 (−0.16, 1.11)	98	MD: 0.26 (−0.06, 0.58) (−1.12, 1.64)	95	55.2	N	0.24	MD: 0.07 (−0.00, 0.14)	3
Palacios et al. (41)	Yuan et al. (76)	Vitamin D	Pre-eclampsia	5	RR: 0.96 (0.65, 1.42)	0/0.00	15	OR: 0.62 (0.50, 0.77)	60/0.10	RR/OR: 0.66 (0.54, 0.81) (0.33, 1.30)	55/0.10	12.2	Y	0.06	RR/OR: 0.67 (0.60, 0.76)	3
Rees et al. (42)	Kastorini et al. (79)	Healthy diet	Systolic blood pressure, mm Hg	11	MD: −2.61 (−3.91, −1.31)	55	1	MD: 0.80 (−0.84, 2.44)	NA	MD: −2.18 (−3.55, −0.82) (−6.19, 1.82)	67	85.9	N	0.001	MD: −1.86 (−2.45, −1.28)	2
Rees et al. (42)	Kastorini et al. (79)	Healthy diet	Diastolic blood pressure, mm Hg	11	MD: −1.45 (−2.22, −0.68)	45	1	MD: 0.90 (−0.38, 2.18)	NA	MD: −1.21 (−2.05, −0.36) (−3.66, 1.25)	61	87.2	N	0.002	MD: −1.03 (−1.46, −0.60)	2
Rees et al. (43)	Jayedi et al. (95)	Selenium	All-cause mortality	2	RR: 0.97 (0.88, 1.08)	0/0.00	3	RR: 0.79 (0.73, 0.85)	0/0.00	RR: 0.86 (0.77, 0.96) (0.60, 1.22)	62/0.01	40.6	Y	0.001	RR: 0.85 (0.80, 0.91)	2
Rees et al. (43)	Xiang et al. (80)	Selenium	Cardiovascular mortality	2	RR: 1.02 (0.74, 1.41)	44/0.03	3	RR: 0.77 (0.63, 0.94)	6/0.00	RR: 0.85 (0.70, 1.04) (0.50, 1.46)	38/0.02	45.5	N	0.15	RR: 0.86 (0.75, 0.98)	3
Rees et al. (43)	Zhang et al. (81)	Selenium	Cardiovascular disease	2	RR: 1.03 (0.95, 1.11)	0/0.00	14	RR: 0.87 (0.76, 1.00)	4/0.00	RR: 0.94 (0.85, 1.04) (0.77, 1.15)	16/0.01	47.2	N	0.04	RR: 0.98 (0.92, 1.05)	3
Rees et al. (44)	Kastorini et al. (79)	Mediterranean diet	HDL, mmol/L	6	MD: 0.02 (−0.01, 0.04)	0	1	MD: 0.01 (−0.04, 0.06)	NA	MD: 0.02 (−0.01, 0.04) (−0.01, 0.04)	0	82.3	N	0.84	MD: 0.02 (−0.01, 0.04)	2
Rees et al. (44)	Kastorini et al. (79)	Mediterranean diet	Triglycerides, mmol/L	7	MD: −0.09 (−0.17, −0.01)	16	1	MD: −0.02 (−0.07, 0.03)	NA	MD: −0.06 (−0.13, −0.00) (−0.19, 0.07)	25	64.0	Y	0.15	MD: −0.05 (−0.09, −0.01)	2
Rees et al. (44)	Kastorini et al. (79)	Mediterranean diet	Systolic blood pressure, mm Hg	4	MD: −1.50 (−3.92, 0.92)	16	1	MD: 0.80 (−0.84, 2.44)	NA	MD: −0.56 (−2.60, 1.48) (−6.14, 5.03)	38	59.7	N	0.12	MD: −0.07 (−1.38, 1.23)	2
Rees et al. (44)	Rosato et al. (83)	Mediterranean diet	Cardiovascular mortality	1	HR: 0.81 (0.50, 1.32)	NA	7	RR: 0.74 (0.67, 0.81)	47/0.01	HR/RR: 0.74 (0.68, 0.81) (0.61, 0.91)	40/0.01	5.4	Y	0.71	RR: 0.78 (0.75, 0.81)	2
Rees et al. (44)	Rosato et al. (83)	Mediterranean diet	Cardiovascular disease	1	HR: 0.70 (0.58, 0.85)	NA	11	RR: 0.81 (0.74, 0.88)	80/0.01	HR/RR: 0.80 (0.74, 0.87) (0.62, 1.03)	78/0.01	10.8	N	0.19	RR: 0.84 (0.82, 0.87)	2
Rees et al. (44)	Soltani et al. (82)	Mediterranean diet	All-cause mortality	1	HR: 1.00 (0.81, 1.24)	NA	26	RR: 0.90 (0.89, 0.92)	80/0.00	HR/RR: 0.91 (0.89, 0.92) (0.86, 0.95)	78/0.00	0.5	Y	0.34	RR: 0.92 (0.92, 0.93)	2
Rutjes et al. (45)	Doets et al. (84)	B-vitamins	Dementia/MCI	1	RR: 1.01 (0.69, 1.48)	NA	3	RR: 0.99 (0.99, 1.00)	22/0.00	RR: 0.99 (0.99, 1.00) (0.98, 1.01)	0/0.00	0.0	N	0.95	RR: 0.99 (0.99, 1.00)	3
Rutjes et al. (45)	Goodwill and Szoeke (85)	Vitamin D	Dementia/MCI	1	RR: 1.09 (0.70, 1.71)	NA	14	OR: 0.88 (0.82, 0.95)	56/0.01	RR/OR: 0.88 (0.82, 0.95) (0.71, 1.11)	54/0.01	2.3	Y	0.34	RR/OR: 0.91 (0.87, 0.95)	3
Tieu et al. (47)	Chia et al. (87)	Healthy diet	Preterm birth	3	RR: 0.52 (0.21, 1.28)	0/0.00	5	OR: 0.81 (0.69, 0.94)	31/0.01	RR/OR: 0.83 (0.75, 0.93) (0.70, 1.00)	6/0.00	1.4	Y	0.35	RR/OR: 0.86 (0.79, 0.93)	2
Tieu et al. (47)	Chia et al. (87)	Healthy diet	Small gestational age	2	RR: 0.84 (0.49, 1.42)	0/0.00	8	OR: 0.88 (0.71, 1.08)	36/0.03	RR/OR: 0.88 (0.75, 1.03) (0.64, 1.21)	19/0.01	8.1	N	0.88	RR/OR: 0.91 (0.86, 0.97)	2
Tieu et al. (47)	Chia et al. (87)	Healthy diet	Birth weight, g	5	MD: 5.94 (−51.11, 62.99)	0	12	MD: −9.61 (−53.12, 33.91)	86	MD: −8.56 (−46.48, 29.36) (−152.77, 135.64)	81	17.2	N	0.67	MD: 36.30 (22.33, 50.26)	2
Tieu et al. (47)	Mijatovic-Vukas et al. (88)	Healthy diet	Gestational diabetes	5	RR: 0.61 (0.36, 1.04)	54/0.18	4	OR: 0.70 (0.62, 0.80)	6/0.00	RR/OR: 0.69 (0.59, 0.81) (0.48, 1.00)	33/0.02	22.4	Y	0.60	RR/OR: 0.71 (0.63, 0.79)	2
Vinceti et al. (19)	Vinceti et al. (19)	Selenium	Cancer	5	RR: 0.99 (0.86, 1.14)	46/0.01	7	OR: 0.72 (0.55, 0.93)	46/0.06	RR/OR: 0.86 (0.73, 1.01) (0.52, 1.42)	64/0.04	54.3	N	0.03	RR/OR: 0.94 (0.88, 1.01)	3
Vinceti et al. (19)	Vinceti et al. (19)	Selenium	Cancer mortality	2	RR: 0.81 (0.49, 1.32)	79/0.10	1	OR: 0.93 (0.83, 1.04)	NA	RR/OR: 0.90 (0.78, 1.05) (0.53, 1.54)	46/0.01	33.6	N	0.58	RR/OR: 0.92 (0.83, 1.01)	2
Vinceti et al. (19)	Vinceti et al. (19)	Selenium	Colorectal cancer	3	RR: 0.74 (0.41, 1.33)	48/0.13	1	OR: 0.80 (0.68, 0.94)	NA	RR/OR: 0.82 (0.64, 1.04) (0.38, 1.78)	28/0.02	40.5	N	0.80	RR/OR: 0.82 (0.71, 0.94)	2
Yao et al. (49)	Aune et al. (91)	Fiber	Colorectal cancer	2	RR: 2.69 (1.06, 6.82)	0/0.00	19	RR: 0.88 (0.82, 0.94)	4/0.00	RR: 0.88 (0.82, 0.96) (0.74, 1.05)	18/0.01	0.7	Y	0.02	RR: 0.88 (0.83, 0.94)	2
Yao et al. (49)	Ben et al. (90)	Fiber	Colorectal adenoma	5	RR: 1.04 (0.94, 1.14)	4/0.00	4	RR: 0.92 (0.76, 1.11)	33/0.01	RR: 1.00 (0.91, 1.11) (0.82, 1.23)	23/0.00	65.9	N	0.26	RR: 1.00 (0.93, 1.08)	3

¹BoE, bodies of evidence; CE, common-effect model; CS, cohort study; MCI, mild cognitive impairment; MD, mean difference; NA, not applicable; PI, prediction interval; PI/ECO, population, intervention/exposure, comparator, outcome; RCT, randomized controlled trial; RE, random-effects model; RR, risk ratio.

FIGURE 1.

Effect of including CSs (in red) on meta-analysis conclusions on diet–disease associations for all-cause mortality. Green colors indicate effect estimates from a meta-analysis restricted to RCTs only. The diamond indicates the effect estimates from a meta-analysis considering all studies (RCTs and CSs). Heterogeneity across studies was assessed with the τ². CS, cohort study; RCT, randomized controlled trial; RR, risk ratio.

FIGURE 2.

Effect of including CSs (in red) on meta-analysis conclusions on diet–disease associations for CVD. Green colors indicate effect estimates from a meta-analysis restricted to RCTs only. The diamond indicates the effect estimates from a meta-analysis considering all studies (RCTs and CSs). Heterogeneity across studies was assessed with the τ². IHD, ischemic heart disease; CS, cohort study; CVD, cardiovascular disease; CVM, cardiovascular mortality; RCT, randomized controlled trial; RR, risk ratio.

Pooling scenarios

By pooling BoE from RCTs and CSs with a random-effects model, for 45 (56%) out of 80 diet–disease associations the 95% CI excluded no effect and showed mainly a reduced risk/inverse association. For the common-effect model, for 52 (65%) out of 80 diet–disease associations the 95% CI excluded no effect. The effect sizes (RR/HR/OR) for dichotomous outcomes were mainly in the range of 0.75–1.25, which could not be considered an effect of large magnitude. The test for subgroup difference comparing BoE from RCTs and BoE of CSs was statistically significant (P < 0.05) for 24 (30%) diet–disease associations. By pooling BoE from RCTs and CSs the median I² was 46% (τ² = 0.01), whereas the mean I² was 43% (τ² = 0.02). The contributed weight of RCTs to the pooled estimates was 34% (median) and 38% (mean). As for the 95% PIs, 11% (n = 9) of the pooled BoE from RCTs and CSs excluded no effect.

The direction of effect between BoE from RCTs and pooled effect estimates was rarely opposite (n = 17; 21%). Discordant direction of effects was mainly attributed to the comparison of micronutrient supplements in BoE of RCTs with dietary micronutrient intake in BoE of CSs (n = 12). The integration of BoE from CSs modified the conclusion from BoE of RCTs in 35 (44%) of the 80 diet–disease associations (i.e., 95% CI excluded no effect changed to 95% CI overlapped no effect or vice versa); in 21 (60%) of these 35 BoE the direction of effect was concordant. In 16 (46%) of these 35 diet–disease associations the test of subgroup difference was statistically significant (P < 0.05) comparing BoE from RCTs and BoE from CSs (in 7 of these 16 associations the direction of effect was opposite). In 9 (26%) of these 35 diet–disease associations the degree of PI/ECO similarity was judged as “broadly similar.” Eighteen (51%) of these diet–disease associations investigated the effects of micronutrient supplements in BoE of RCTs, compared with dietary micronutrient intake in BoE of CSs.

Discussion

Summary of findings

As far as we know, this is the first empirical study evaluating the impact scenario of pooling BoE from RCTs and CSs in nutrition research. Overall, 160 effect estimates based on 773 RCTs and 720 CSs were analyzed. By pooling BoE from RCTs and CSs, in ∼60% of the diet–disease associations the 95% CI excluded no effect, whereas in ∼20% of the included BoE from RCTs the 95% CI excluded no effect. The test for subgroup difference comparing BoE from RCTs and BoE of CSs was statistically significant for 30% of pooled estimates. The contributed weight of BoE from RCTs to the pooled estimates was 34%, showing clearly that BoE of CSs were the main evidence contributor in our study. This had an important influence on the degree of statistical heterogeneity, for example the median I² and τ² for the pooled estimates were 46% and 0.01, respectively (I² = 0%, τ² = 0 in BoE of RCTs; I² = 55%, τ² = 0.01 in BoE of CSs). The integration of BoE from CSs modified the conclusion derived from BoE of RCTs in nearly 50% of the diet–disease associations. However, the direction of effect between BoE of RCTs and pooled estimates was mainly concordant, suggesting that statistical precision increased substantially by adding evidence from CSs.

Comparison with other studies

We could not identify any similar empirical study using a pooling scenario of different study designs in the field of medical research. A recent study of 102 therapeutic MAs showed that in 38% of MAs both observational studies and RCTs were combined in a single MA without subgroups. In 15% of cases they were evaluated together but with a subgroup analysis, in 20% of cases they were pooled separately, and in 27% of cases only RCTs were pooled with a qualitative description of observational studies (96). In most cases a random-effects model was used and the integration of observational studies was not justified by most authors. When comparing results of MAs including both BoE (combined without a subgroup) and MAs restricted to RCTs only, the conclusion was modified by the integration of observational studies for nearly 71%. In our study adding evidence from CSs, the conclusion from BoE of RCTs was modified for 44% of the included diet–disease associations but the direction of effect was mainly concordant. However, especially associations very close to the null should be interpreted with caution, because the pooled results may be a function of bias and/or confounding, and not necessarily a true association.

In the methodological study by Bun et al. (96) it was also shown that MAs of both BoE (with subgroups) indicated no modification of the conclusion. In line with our findings, the authors found that including observational studies frequently increased statistical heterogeneity. Therefore, they recommended analyzing RCTs and observational studies in separate MAs and suggested improving justifications for including observational studies in MAs. Another study comparing effects of interventions based on observational studies and RCTs with regard to 3 clinical topics showed that effects were similar (97). Anglemyer et al. (98) found little evidence for significant effect estimate differences between observational studies and RCTs. Nevertheless, they stated that the lack of difference in effect estimates does not imply that RCTs and observational studies can be pooled because there are situations in which estimates greatly differ. The latter situation could be subject to further research. Therefore, they recommended analyzing RCTs and observational studies in separate MAs.

Implications that follow for the research nutrition field

There has been a long debate regarding what constitutes best evidence in nutrition research, and whether it emerges from RCTs. RCTs are considered the ideal methodology for causal inference and in which the effects of a dietary change on disease or intermediate disease markers are evaluated (99). However, most dietary intervention RCTs are of short duration and often do not target patient-relevant outcomes such as morbidity or mortality. Cohort studies, on the other hand, provide less robust information regarding causality, but are usually considered more applicable for nutrition research (100).

In the present study, the median contributed weight of BoE of RCTs to the pooled estimates was smaller (34%) than for BoE from CSs (66%). These weights are highly dependent on sample size (for dichotomous or continuous outcomes) and number of events (for dichotomous outcomes), which were often lower across BoE of RCTs. Given that most evidence on diet–disease associations is based on observational studies, this finding was not unexpected. However, we also identified several diet–disease associations in which weights of RCTs were higher (e.g., omega-3 fatty acids and mortality, β-carotene and mortality, sodium and blood pressure, vitamin D and fracture risk).

Because BoE from CSs can complement BoE from RCTs, and vice versa, as shown in our study, clear guidance for integration of both BoE in nutrition evidence syntheses is greatly needed. Similar to our findings, a cross-sectional study has shown that only very few Cochrane nutrition reviews (2%) include observational studies (6), which has been criticized already in the past (7). Therefore, we recommend in line with other authors that CSs should be considered for inclusion in future Cochrane nutrition reviews (6).

Implications that follow for the broader research field

In a survey investigating the rationale, perceptions, and preferences for the integration of RCTs and nonrandomized studies of interventions (NRSI) in evidence syntheses, Cuello-Garcia et al. (101) showed that the most frequent approach was to conduct separate MAs for RCTs and NRSI. However, nearly half of the experts interviewed, on ≥1 occasion, pooled RCTs and NRSI in MAs (29% via subgroup, and 18% in a single MA).

Turner et al. (102) investigated statistical heterogeneity in nearly 15,000 MAs including ∼2000 Cochrane reviews and observed for objective outcomes a median τ² between 0.01 and 0.02, which was similar to our findings. In line with our findings, the Cochrane Handbook indicated that authors should expect greater statistical heterogeneity in an SR of NRSI than in an SR of RCTs. Reasons include the diverse ways in which NRSI may be designed to investigate the effects of interventions/exposures, partly due to the increased potential for methodological variation between primary studies, and the resulting variation in their risk of bias (e.g., measuring exposure and outcome, or adjustment for more or fewer important confounding domains). The Cochrane Handbook recommends that review authors should exclude from analysis any NRSI judged to be at critical RoB and may choose to include only studies that are at moderate or low RoB, specifying this choice a priori in the review protocol (103). The handbook recommends that RCTs and NRSI should not be combined in an MA [although the power to detect an effect may increase (104)], and that for example CSs and case-control studies should not be combined in an MA if they address different research questions. Given that heterogeneity between NRSI is expected to be high because of their diversity, the random-effects MA approach should be the default choice. In a methodological survey on the use of the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach for rating the certainty of evidence in nutrition evidence syntheses, we showed recently that downgrading for inconsistency was more common in SRs of observational studies (29%) than in SRs of RCTs (15.1%) (105). Guidance on the decision regarding when to search for and include either or both types of studies in SRs has been recently published by the GRADE Working Group (106).

In contrast to the recommendations of Cochrane, in a recent framework for the synthesis of NRSI and RCTs, the pooling of both study designs is not opposed in principle (107). Moreover, a scoping review of 93 articles, summarizing the methods to systematically review and meta-analyze observational studies, highlighted that existing guidance is highly conflicting for pooling if results are similar over different study designs (108). Finally, in several high–impact factor journal MAs, both study designs were pooled (109–112). Overall, it looks like this is a gray area that needs further methodological research, because a comprehensive guidance document on how to pool both BoE is lacking (108).

Strengths and limitations

This study has several strengths. First, no similar study has been conducted so far. Second, we analyzed a large sample of diet–disease pairs (n = 80; based on 160 pooled estimates), which was based on >700 RCTs and >700 CSs; both study designs are considered as the most trustworthy in nutrition research (5). Third, we selected BoE of RCTs published as Cochrane reviews, which are internationally recognized as the highest standard in evidence-based health care. The high methodological quality of Cochrane nutrition reviews has been confirmed (6). Fourth, our study was based on MAs of binary outcomes, and also continuous outcomes.

Limitations of this study are as follows. First, although we pooled a large sample of diet–disease associations, our sample may not be representative of all MAs, and the totality and most updated evidence of available diet–disease associations might provide different results. Second, we pooled BoE from RCTs derived from Cochrane reviews with BoE derived from CSs (non-Cochrane reviews), and pooling of these 2 study designs/publications within a single SR of both RCTs and CSs might provide different results. Overall, 9 (20%) out of 46 included SRs of CSs included also RCTs, but MAs were performed for different outcomes, and only 6% of the included Cochrane reviews included also CSs. Third, we did not consider or weight RoB of primary studies in our pooling scenario. Fourth, no diet–disease association was judged as “more or less identical,” indicating that BoE of RCTs and CSs differ at least slightly in terms of PI/ECO criteria and caution is therefore required when pooling both BoE. Fifth, the potential for confounding in the individual cohort studies and subgroup analyses in the MA cannot be ruled out. Several subgroups also included only a small number of studies. Sixth, particularly for the BoE from CSs, some CSs were included multiple times, and from the SRs, the same original studies were used with the same exposure but for different outcomes. Because of these limitations, and the fact that causal effects of diet cannot be determined in MAs of cohort studies, our findings need to be interpreted with caution.

Conclusion

This large pooling scenario study showed that the integration of BoE from CSs modified the conclusion from BoE of RCTs in nearly 50% of included diet–disease associations, although the direction of effect was mainly concordant between BoE of RCTs and pooled estimates. The median contribution weight of RCTs to the pooled estimates was 34%, and the statistical inconsistency was substantially driven by integrating BoE from CSs. Our findings provide a first insight regarding the potential impact of pooling both BoE in prospective nutrition evidence syntheses. Because only very few Cochrane nutrition reviews include CSs, and most evidence in nutrition research comes from CSs, there is urgent need for evidence-based guidance for the potential integration of both BoE—not only for nutrition evidence syntheses, because a comprehensive guidance document is lacking. In line with other authors, we recommend at this stage analyzing RCTs and CSs in separate MAs, or, if combined together, with a subgroup analysis, a random-effects model, and excluding CSs with a critical RoB."

Notes

Supported by Deutsche Forschungsgemeinschaft (DFG; German Research Foundation) Projektnummer 459430615 (to LS) and Forschungskommission der Medizinischen Fakultät Freiburg. The funders had no role in considering the study design or in the collection, analysis, and interpretation of data, writing of the report, or decision to submit the manuscript for publication.

Author disclosures: LS is a member of the Editorial Board of Advances in Nutrition and a member of the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) working group. JJM is a member of the GRADE Guidance group. All other authors report no conflicts of interest.

Supplemental Appendices A and B, Supplemental Tables 1–4, and Supplemental Figures 1–80 are available from the “Supplementary data” link in the online posting of the article and from the same link in the online table of contents at https://academic.oup.com/advances/.

Abbreviations used: BoE, bodies of evidence; CS, cohort study; GRADE, Grading of Recommendations, Assessment, Development and Evaluations; MA, meta-analysis; MD, mean difference; NCD, noncommunicable disease; NRSI, nonrandomized studies of interventions; PI, prediction interval; PI/ECO, patients/population, intervention/exposure, comparator, and outcome; RCT, randomized controlled trial; RoB, risk of bias; RR, risk ratio; SR, systematic review.