Reply to Nuijten et al.: Reanalyses actually confirm that US studies overestimate effects in softer research

We thank Nuijten et al. (1) for their use of our data. Their reanalyses show patterns that clearly match our original findings (2), but their inferences are weakened by their choice of statistical assumptions.

In our report (2), effect sizes were all rescaled around the summary estimate of each meta-analysis, and we analyzed primary studies as a population of deviation scores, nested by meta-analysis. Instead, Nuitjen et al. synthesized unscaled metaregression coefficients (1). Lack of scaling is problematic because the topics considered in the meta-analyses are very different and distributions of unscaled effects have several outliers.

Nuitjen et al. back their claims of no “US effect” by plotting datapoints in histograms, but this violates the very essence of meta-analysis, which is to weight datapoints by an inverse function of their SE. Moreover, in our study we only observed the US effect among behavioral studies, so graphs should be partitioned by method.

Using Nuijten et al.’s (1) R code, we plotted their regression estimates against the respective SEs. The “heavier” datapoints (those with small SEs) tend to be more positive for behavioral studies compared with nonbehavioral (Fig. 1, Left). This is essentially the same pattern that we described in our original analysis. Indeed, Nuijten et al. estimated US effects identical in direction to those we had described, and the difference between behavioral and nonbehavioral meta-analyses is even close to formal statistical significance (Table 1, first column). The only correction we introduced to Nuijten et al.’s code was to assume the same random-effects estimation method (i.e., DerSimonian and Laird) at both levels of analysis. Their original R code used a different estimator at the upper level (in the metameta-analysis), a choice that reduced statistical significance with no apparent justification.

Fig. 1.

Metaregression values of US effect [ values calculated without the US_ijSE_ij interaction, as in Nuijten et al. (1)] plotted against their respective SEs (SE_βUS,) (see text for details).

Table 1.

Magnitude of the US effect with different model assumptions for level 1 and level 2

	Level 2	Random	Fixed	Random
	Level 1	Random	Fixed	Fixed
βUS, NB		−0.052 ± 0.028 (0.063)	−0.047 ± 0.02 (0.021)	−0.043 ± 0.026 (0.094)
βUS, BB		0.158 ± 0.078 (0.042)	0.19 ± 0.052 (<0.001)	0.169 ± 0.089 (0.057)
βUS, BE		0.057 ± 0.072 (0.42)	0.123 ± 0.035 (<0.001)	0.185 ± 0.083 (0.025)
βUS, (BB + BE) vs. NB		0.103 ± 0.058 (0.078)	0.141 ± 0.032 (<0.001)	0.178 ± 0.069 (0.01)

Values are metameta-analytical estimates ± standard error (P value), obtained in separate analyses. BB, bio-behavioral meta-analyses; BE, purely behavioral meta-analyses; NB, non-behavioral meta-analyses. For details see text and ref. 1.

Nuijten et al.’s (1) assumption of random effects at two levels, however, makes everything fluctuate randomly, even within the group of similar studies from the same country in the same meta-analysis. Random effects yield unstable results and very wide uncertainty for meta-analyses with relatively small numbers. Not surprisingly, formal significance is lost when these regression coefficients are synthesized.

To further illustrate the impact of analytical choices in the Nuitjen et al. (1) approach, we calculated regression estimates assuming fixed effects. Most of them are positive among behavioral meta-analyses (Fig. 1, Right), and a metameta-analysis using fixed effects at two levels would actually show an extremely strong US effect (Table 1, second column). However, an assumption of fixed effects at all levels (equivalent to assuming that the US effect is identical in all circumstances) is probably unrealistic. The magnitude of the US effect may be expected to vary across fields, and this is expressed by assuming random effects between meta-analyses (i.e., at level 2). If analyzed with this assumption, Nujten et al.’s calculations would show a US effect very similar to what we originally described (Table 1, third column). Therefore, Nuitjen et al.’s reanalyses actually verify our conclusions. Nevertheless, additional studies should be encouraged to probe for the potential presence and magnitude of US effects also in other disciplines and larger datasets.