Concordance between nutritional cohorts and randomized trials: biological confirmation or statistical consequence? A re-analysis of Stadelmaier et al.

Abstract

Background

Stadelmaier et al. recently reported a pooled ratio of risk ratios (RRR) of 1.00 between population, intervention/exposure, comparator, outcome (PI/ECO) matched randomized controlled trials (RCTs) and cohort studies, interpreting this as evidence that randomized trials “confirm” observational findings in nutrition with “tremendous public health implications.” We hypothesized that this apparent agreement is not a validation of concordance, but an expected statistical effect arising from the pooling of null or small effect sizes with large variances rather than pairwise concordance.

Methods

We re-analyzed the authors’ binary dataset (n = 54 pairs) using a permutation framework (B = 10,000) in which cohort studies were randomly reassigned to RCTs, breaking the original pairing (both globally and restricted within exposure-type strata) while preserving marginal distributions.

Results

Under random pairing, 99.96% of permuted pooled RRRs fell within the authors’ reported confidence interval [0.91,1.10], and the observed RRR was statistically indistinguishable from the random-pairing permutation distribution (p = 0.12). A pairwise discrepancy statistic showed no improvement with matching over random pairing (p = 0.21). A precision-weighted statistic was significant (p = 0.008), but weight was highly concentrated: 4 pairs (7% of n) accounted for 54.8% of total weight, and the effective number of pairs was 9.9 (18% of n). Qualitative concordance using the authors’ prior criteria was only 13% (7/54 pairs).

Conclusions

A pooled RRR near 1.00 indicates only that cohort effect sizes are not systematically different from RCT estimates. In this dataset, an RRR near 1 is a result expected under random pairing. Pairwise agreement is not improved by PI/ECO matching for typical comparisons; the significant weighted statistic reflects alignment among a small minority of high-precision pairs, not “most” findings. Pooled RRR lacks resolution as a confirmation metric in fields with small effects and should not be interpreted as evidence of replication; otherwise, it may inappropriately influence evidence synthesis practices.

Background

We commend Stadelmaier et al. for their ambitious meta-epidemiological study [1] in which the authors matched individual RCT-cohort pairs on population, intervention/exposure, comparator, and outcome (PI/ECO) characteristics and reported that PI/ECO similarity was “convincing.” For the 54 binary-outcome pairs, they reported a pooled ratio of risk ratios (RRR) of 1.00 (95% CI 0.91–1.10) and concluded that “our study shows clearly that most findings from individual RCTs were confirmed in best-matching cohort studies,” adding that “the public health implications of our findings are therefore tremendous.”

In our view, there is a mismatch between the question the authors sought to answer and the question they addressed. Their stated aim was “to evaluate the agreement of effect estimates from individual nutrition RCTs and cohort studies investigating a highly similar research question,” which implies an assessment of pairwise correspondence; that is, whether matched RCT-cohort pairs yield concordant estimates at the level of individual comparisons. However, the operational definition of “agreement” was meta-analytic pooling: “Agreement of matched RCTs/cohort studies was analyzed by pooling ratio of risk ratios.” A pooled RRR near 1.00 simply shows whether there is systematic average deviation between study designs after inverse-variance weighting; it does not measure within-pair agreement nor establish how many pairs agree.

Since, in nutrition research, small effects and large variances are common, a pooled RRR near one can arise through sign cancellation and precision-weighting even when many matched pairs differ materially in magnitude or direction. Accordingly, a pooled RRR compatible with substantial pairwise discordance cannot, by itself, support claims that “most” individual comparisons are confirmed. If the aim is to evaluate agreement between individual RCTs and cohort studies, a better question is whether PI/ECO matching improves pairwise alignment beyond what would be expected under negative-control random re-pairing.

In this paper, we used a permutation test (B = 10,000) to compare the observed matched dataset to randomly paired RCT-cohort combinations. For the authors’ aimed question, we tested whether matched pairs show closer pairwise agreement than random pairing using within-pair discrepancy metrics. For the authors’ answered question, we assessed whether the observed pooled RRR is informative by situating it within the permutation distribution expected under the null. Finally, we assessed qualitative concordance using the authors’ prior criteria.

Methods

For each of the n = 54 matched RCT-cohort pairs reported by Stadelmaier et al., we extracted the reported relative risk (RR) and corresponding 95% confidence intervals (CI) for both the RCT and cohort estimates. All analyses were conducted on the log scale.

Pairwise discrepancy metrics

For each matched pair i, we defined the discrepancy as:

$${\mathrm{\Delta }}_i=\text{log}\left({\mathit{RRR}}_i\right)=\text{log}({\mathit{RR}}_{\mathit{RCT},i})-\text{log}\left({\mathit{RR}}_{\mathit{COH},i}\right)$$ 1

and used the absolute discrepancy |${\mathrm{\Delta }}_i$| as our measure of disagreement. Our primary test statistic, which was chosen due to its capacity to reflect typical pairwise alignment while being robust to extreme values, was the median absolute discrepancy across pairs:

$$T_{\mathit{median}}=median(\left|{\mathrm{\Delta }}_i\right|)$$ 2

Precision-weighted and uncertainty sensitivity analysis

As a sensitivity analysis, to evaluate whether conclusions depended on the relative precision of individual comparisons, we derived standard errors from reported 95% CIs as:

$$\mathit{SE}=\frac{\text{log}\left({\mathit{CI}}_{\mathit{upper}}\right)-\text{log}({\mathit{CI}}_{\mathit{lower}})}{2\times 1.96}$$ 3

and we defined inverse-variance weights as:

$$w_i=\frac{1}{{\mathit{SE}}_{\mathit{RCT},i}^2+{\mathit{SE}}_{\mathit{COH},i}^2}$$ 4

Therefore, our precision-weighted statistic is:

$$T_{w\mathit{mean}}=\frac{{\sum }_{i=1}^n{w}_i\left|{\mathrm{\Delta }}_i\right|}{{\sum }_{i=1}^n{w}_i}$$ 5

In our sensitivity analyses, we sought to evaluate the possible discordance between $T_{w\mathit{mean}}$, and the primary outcome $T_{\mathit{median}}$.To do so, we ranked pairs by weight to demonstrate their cumulative weight contribution and to get the effective number of pairs driving the original result. For this purpose, a Kish effective sample size is defined as:

$$n_{\mathit{eff}}=\frac{{(\sum w_i)}^2}{\sum w_i^2}$$ 6

To account for uncertainty, we calculated a standardized pairwise discrepancy as:

$$z_i=\frac{|\text{log}\left({\mathit{RRR}}_i\right)|}{\sqrt{{\mathit{SE}}_{\mathit{RCT},i}^2+{\mathit{SE}}_{\mathit{COH},i}^2}}$$ 7

and summarized it with its median and the proportion in which $z<1.96$.

Finally, we compared these summaries to their permutation distributions to obtain one-sided permutation p-values. Therefore, a smaller median z and a larger proportion in which z < 1.96 indicate greater agreement.

Permutation procedure

We tested the null hypothesis (that PI/ECO matching does not provide a pairwise agreement beyond chance) by generating 10,000 random shuffles (permutations) of the cohort estimates across pairs while keeping RCT estimates fixed, thereby breaking the original matching structure.

For each permutation b, the cohort’s log-RR and its associated SE were reassigned as a unit, thus preserving the pairing between effect size and precision. Permuted discrepancies were computed as:

$${\mathrm{\Delta }}_i^{\left(b\right)}=\text{log}({\mathit{RR}}_{\mathit{RCT},i})-\text{log}\left({\mathit{RR}}_{\mathit{COH},{\pi }_b\left(i\right)}\right)$$ 8

where ${\pi }_b$ is a random shuffle of $\left\{1,,,\cdots ,,,n\right\}$, and then calculated $T_{\text{median}}^{(b)}$ and $T_{w\text{mean}}^{(b)}$ from $\left|{\mathrm{\Delta }}_i^{\left(b\right)}\right|$.

Because smaller values of $T_{\text{median}}$ and $T_{w\text{mean}}$ indicate better agreement, we computed one-sided permutation p-values as:

$$p=\frac{1+{\sum }_{b=1}^B\mathbf{1}\{T^{(b)}\le T_{\mathit{obs}}\}}{B+1}$$ 9

Pooled RRR analysis

To characterize how the observed pooled RRR compares with what would be expected if cohort estimates were exchangeable across RCTs, we first performed an unrestricted permutation analysis as it was described above. For each permuted dataset, we recomputed the pooled RRR using the authors’ original approach (random-effects meta-analysis with the Paule-Mandel (PM) ${\tau }^2$ estimator), yielding an empirical reference distribution of pooled RRRs against which the observed pooled RRR was compared via medians and percentiles.

As a sensitivity analysis, we repeated the permutation procedure using restricted stratified reassignment within exposure (supplement vs intake) × PI/ECO strata to preserve key design features and relax global exchangeability assumptions. Two-sided p-values tested whether the observed pooled RRR was extreme relative to the random-pairing distribution, and were calculated as:

$$p=2\text{min}(q,1-q)$$ 10

where:

$$q=\frac{1+{\sum }_{b=1}^B\mathbf{1}\{{\text{log}(\mathit{RRR}}^{\left(b\right)})\le {\text{log}(\mathit{RRR}}_{\mathit{obs}})\}}{B+1}$$ 11

Qualitative concordance

We calculated the qualitative concordance using the authors’ methodology from prior work. Qualitative concordance is defined as observing (i) statistically significant effects in the same direction across designs, or (ii) both effect sizes being non-significant with 95% CIs entirely within 0.80–1.25.

All statistical analyses were performed using R version 4.5.2 (with RStudio version 2026.01.0), and all the R packages were updated to their latest version (09 February 2026).

Results

Pooled RRR under random pairing

Using the original matched cohort-RCT pairs and the authors’ methodology of random-effects with PM ${\tau }^2$ estimator, the observed pooled RRR was 1.00 (95% CI: 0.91–1.10), replicating the pooled estimate reported by Stadelmaier et al.

For the random permutation procedure, the median pooled RRR was 1.03 (95% permutation percentile interval 0.99–1.07), with 99.96% of the 10,000 randomly paired datasets producing pooled RRRs within the 95% CI range reported by the authors (0.91–1.10) (Fig. 1A).

Fig. 1.

A Pooled RRR under random pairing using PM. Red line: observed data. Green dashed: Stadelmaier et al. [1] CI [0.91,1.10]. 99.96% (9996/10,000) of pooled RRR permutations within the originally reported 95% CI [0.91–1.10]. B Primary statistic $T_{\text{median}}$ distribution under random pairing. Red line: observed data. One-sided p = 0.21. C Sensitivity statistic $T_{w\text{mean}}$ under random pairing. Red line: observed data (0.11). One-sided p = 0.008. Number of permutations for all figures B = 10,000

The observed median |∆| was not statistically distinguishable from the random-pairing permutation distribution (p = 0.12) (Table 1 (1)). Crucially, to ensure our null distribution was not artificially inflated by clinically implausible pairings, we conducted a sensitivity analysis restricting permutations within exposure × PI/ECO strata. This enforced exchangeability yielded similar results (median permuted pooled RRR 1.01, 95% permutation interval 0.98–1.05) and rendered the observed RRR statistically indistinguishable from the null (p = 0.48) (Table 1 (2)).

Table 1.

Comparison of the observed matched data with B = 10,000 random permutations. For pairwise metrics, observed ∆ is the discrepancy on the log-RRR scale; |∆| is the absolute discrepancy. ${\text{e}}^{|\mathrm{\Delta }|}$ is the corresponding fold-difference (e.g., 1.31 = 31% difference between RCT and cohort estimates). $T_{\text{median}}$ is the median absolute discrepancy; $T_{\text{wmean}}$ is the inverse-variance weighted mean discrepancy. Permutation p-values for $T_{\text{median}}$ and $T_{\text{wmean}}$ are one-sided (smaller observed values indicate greater agreement). Pooled RRR p-values are two-sided, testing whether the observed pooled estimate differs from the permutation distribution

Pairwise agreement under observed matching

The median absolute discrepancy ($T_{\text{median}}$) among the 54 matched pairs was 0.27 on the log-RRR scale, corresponding to a typical absolute ratio between the cohort and RCT RR estimates of approximately 1.31. This did not differ significantly from the permutation median of 0.31 (one-sided p = 0.21) (Fig. 1B). By this metric, PI/ECO matching did not yield closer pairwise alignment than random re-pairing.

Precision-weighted analysis

The precision-weighted mean discrepancy ($T_{w\text{mean}}$) was significantly smaller under observed matching than is expected by chance (0.11 observed vs. 0.17 permutation; p = 0.008) (Fig. 1C). However, statistical weight was highly concentrated with the top 4 pairs (7% of n = 54) accounting for 54.8% of total inverse-variance weight, and with an effective number of pairs of 9.9 (18% of n).

As is illustrated in Fig. 2, among high-precision pairs (top tertile), 67% were aligned within ± 25%, compared to only 22% among low-precision pairs (bottom tertile). The bottom tertile (18 pairs) only accounted for 1.4%. In our sensitivity analysis, after accounting for uncertainty, the observed median z was 0.92, and 83.3% had z < 1.96; neither differed from random pairing (permuted median z 1.05 (95% perm interval 0.79–1.33); p = 0.17 and a permuted median portion with z < 1.96 of 81.5% (72.2–89%; p = 0.39).

Fig. 2.

Cumulative weight distribution by pair rank. Pairs ranked by precision (1 = most precise). The top 4 pairs (7%) account for 54.8% of the total weight. Points colored by alignment status (green = within ± 25%; red = not aligned)

Qualitative concordance

Applying the qualitative concordance criteria used in the authors’ prior research to the 2025 dataset, we obtained a 13% (7/54 pairs) concordance rate. This low concordance is consistent with our permutation findings that typical pairwise discrepancies are only minimally improved by matching.

Discussion

Our analysis reveals a fundamental mismatch between the statistical properties of pooled RRR and its interpretation as evidence of pairwise confirmation since the pooled RRR and its CI do not distinguish the matched pairing from negative-control random re-pairing in this dataset (p = 0.12). Moreover, the observed interval [0.91, 1.10] is so wide that it contains 9996/10,000 pooled RRRs obtained after randomly re-pairing cohorts to trials; thus, it provides little discrimination between the matched dataset and negative-control re-pairings in this setting.

Our findings do not invalidate the authors’ calculation; rather, they reveal that pooled RRR lacks discriminatory power in fields characterized by small effect sizes and large variances, such as nutritional epidemiology. A possible critique of our primary permutation test is that unrestricted random pairing violates clinical exchangeability. However, our stratified sensitivity analysis, restricted to exposure × PI/ECO strata, preserved key design features and forced plausible comparisons. Under these strict exchangeability constraints, the results remained statistically indistinguishable from chance (p = 0.48), proving that our conclusions do not hinge on global exchangeability assumptions. Separately, our uncertainty sensitivity analysis using standardized z-scores further confirmed the lack of pairwise agreement.

Results challenge the interpretation and tone of the results by Stadelmaier et al. Read literally, “most findings…confirmed” is a majority claim about the set of individual matched comparisons; it is not a statement about a single pooled average. If the intended claim was instead limited to “the pooled RRR is near 1,” then the confirmatory phrasing (“most… confirmed”) overstates what the statistic can support. The pairwise discrepancy statistics reinforce this conclusion. The typical matched pair exhibited a 31% difference between RCT and cohort estimates ($T_{\text{median}}$=0.27), not significantly closer than random pairing (p = 0.21). Consequently, these results are inconsistent with the claim that “most findings from individual RCTs were confirmed by cohort studies.”

On the other hand, while $T_{\mathit{wmean}}$ was statistically significant (p = 0.008), this result is driven by an extreme concentration of statistical weight, with just four pairs (7% of n = 54) accounting for 54.8% of total inverse-variance weight. Meanwhile, the bottom tertile of pairs (n = 18) accounts for merely 1.4% of total weight. The fact that the effective number of pairs, a measure of weight concentration analogous to effective sample size, was 9.9, means that $T_{w\text{mean}}$ behaves as if the dataset contained only ~ 10 equally weighted comparisons rather than 54. This concentration pattern explains the apparent paradox of significant precision-weighted $T_{w\text{mean}}$ alongside the non-significant unweighted $T_{\text{median}}$: The weighted statistic is dominated by a small subset of exceptionally precise comparisons, whereas the unweighted statistic reflects the typical pair.

Put together, these observations show that a significant $T_{w\text{mean}}$ can maybe support concordance for a small subset of high-precision pairs but it cannot justify statements about “most” of the 54 comparisons. Moreover, and in addition to any true concordance within selected pairs, two additional features limit generalizability to the broader set of RCT-cohort comparisons:

• First, two of the four highest-weight pairs (calcium plus vitamin D and fractures, and vitamin E and prostate cancer) are not typical dietary exposure contrasts in nutritional epidemiology; they involve supplementation interventions. For example, the calcium-fracture pair comprised the Women’s Health Initiative (WHI) RCT matched to the WHI Observational Study, where the cohort exposure was supplementation (use vs. non-use) of calcium and vitamin D rather than total dietary calcium intake, which is typically assessed by food frequency questionnaires or short-term recalls.

• Secondly, precision-weighted concordance is highly sensitive to analytic choices that alter inverse-variance weights, particularly when many RCT estimates are near-null. For example, in the vitamin D-breast cancer pair, converting the cohort estimate from a highest-versus-lowest contrast (RR 0.89, 95% CI 0.74–1.08) to a dose–response RR at the RCT dose (RR 0.96, 0.91–1.01 per 400 IU/day) yields near-perfect numerical alignment with the matched RCT (RR 0.96, 0.85–1.09). However, this transformation inherently narrows the confidence interval, producing a 13.2-fold increase in inverse-variance weight (from 107.4 to 1412.4) relative to the original contrast. Likewise, harmonizing the vitamin E-prostate cancer pair shifts the cohort weight from 410.2 to 4273.5, a 10.4-fold increase. While dose harmonization (e.g., Longnecker and Greenland) is a standard and appropriate epidemiological tool for aligning exposure contrasts, its interaction with inverse-variance weighting has profound statistical consequences for measuring agreement. By compressing variance, harmonization inadvertently concentrates statistical weight onto a few near-null contrasts. In this context, a significant precision-weighted metric ($T_{w\text{mean}}$) does not demonstrate that “most” findings agree; rather, it highlights how a few heavily weighted, harmonized null findings can dominate the pooled estimate.

Our qualitative concordance analysis corroborates these findings. Applying the authors’ criteria from prior publications, only 13% (7/54) of pairs were concordant. This continues a declining trend from 24% concordance in the 2022 dataset and 19% concordance in the 2024 dataset [2, 3]. This finding is also consistent with our permutation-based findings.

Beyond statistical considerations, comparing RCT and cohort effect estimates based on PI/ECO matching has conceptual limitations. PI/ECO does not yield complete information regarding the causal estimand of the study, limiting the appropriateness of any comparisons between RCTs and observational studies based solely on PI/ECO [4]. Formal comparisons between RCTs and observational analyses (i.e., benchmarking) generally require harmonizing study protocols, strong assumptions such as no trial engagement effects (which is generally unwarranted in dietary interventions), and often re-analyses of the RCT data to make sure both studies are analyzed in the same way [5, 6].

For example, two studies with the same PI/ECO components will have different causal estimands if they contain different versions of the intervention or the exposure (which is likely in Nutrition studies [7]), if they evaluate different causal contrasts (i.e., intention-to-treat, per-protocol, effect of the treatment strategy), and/or have diverging statistical analyses plans (among other possible discrepancies). It is thus misleading to compare the RR of two studies that targeted a different causal estimand, even if both results were null.

Our analysis has limitations. We relied on the authors’ extracted data without independent verification. We analyzed only binary outcomes (n = 54), and we did not examine continuous pairs. Nonetheless, the methodological implications extend beyond this dataset because pooled RRR is designed to detect systematic direction bias between study designs, which is a valuable function when that is the research question. However, it should not be interpreted as a measure of pairwise confirmation, particularly when effect sizes cluster near the null.

When the aim is “to evaluate the agreement of effect estimates from individual nutrition RCTs and cohort studies investigating a highly similar research question,” explicit evaluation of within-pair discrepancies benchmarked against negative controls is required because conflating the absence of systematic bias with meaningful replication risks overstating concordance and may inappropriately influence evidence synthesis practices.

Conclusions

At most, a pooled RRR near 1 indicates that, on average, nutritional cohort estimates are not systematically larger or smaller than RCT estimates; it does not, by itself, establish pairwise confirmation of individual findings. In our negative-control permutations, a pooled RRR near 1 arose under random re-pairing in 99.96% of 10,000 iterations, showing that the observed pooled result is not distinctive evidence of concordance attributable to PI/ECO matching. When concordance is evaluated at the level implied by “most findings from individual RCTs were confirmed,” the matched data do not support that claim: typical within-pair discrepancies are substantial (median discrepancy approximately 31% by our prespecified discrepancy metric), PI/ECO matching does not improve typical alignment over chance, and qualitative concordance under the authors’ prior criteria is low (13% of pairs).

The statistically significant precision-weighted result is driven by a small subset of 4–10 high-precision comparisons rather than the majority of the 54 pairs. At most, the authors’ results support “similar on average,” not “most confirmed.” The policy-relevant question implied by “tremendous public health implications” is whether such matched cohort estimates can be treated as reliably corroborative of RCT findings; this is not established by a pooled RRR near 1. Treating a pooled RRR as evidence of replication risks conflating a statistical property of pooling with pairwise agreement and may mislead evidence synthesis.

Abbreviations

RR

Relative risk

PM

Paule-Mandel

RRR

Ratio of risk ratios

PI/ECO

Population, intervention/exposure, comparator, outcome

RCT

Randomized controlled trial

Authors’ contributions

MC analysed the dataset. ASM and IN assisted with the analysis design. All authors contributed to interpreting the results, writing the manuscript, and approving this version. All authors read and approved the final manuscript.

Funding

The authors did not receive funding for this work.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

Due to the nature of this work, no recruitment nor IRB approval was needed.

Consent for publication

All authors consent to publish this work under this Journal’s rules.

Competing interests

The authors declare no competing interests.