The assessment of replicability using the sum of p-values

Abstract

Statistical significance of both the original and the replication study is a commonly used criterion to assess replication attempts, also known as the two-trials rule in drug development. However, replication studies are sometimes conducted although the original study is non-significant, in which case Type-I error rate control across both studies is no longer guaranteed. We propose an alternative method to assess replicability using the sum of $p$ -values from the two studies. The approach provides a combined $p$ -value and can be calibrated to control the overall Type-I error rate at the same level as the two-trials rule but allows for replication success even if the original study is non-significant. The unweighted version requires a less restrictive level of significance at replication if the original study is already convincing which facilitates sample size reductions of up to 10%. Downweighting the original study accounts for possible bias and requires a more stringent significance level and larger sample sizes at replication. Data from four large-scale replication projects are used to illustrate and compare the proposed method with the two-trials rule, meta-analysis and Fisher’s combination method.

1. Introduction

Replication studies are increasingly conducted in various fields to assess the replicability of original findings. While ‘replicability’ is intuitively understood as the ability to obtain consistent results when a study is repeated with new subjects [1], there is no agreed-upon quantitative definition of replicability or ‘replication success’. In practice, a variety of different approaches are used that capture different intuitions about what it means for a replication to be successful. For example, metrics such as the ${\displaystyle Q}$ -test or prediction intervals quantify the statistical compatibility of the effect estimates from original and replication studies [2,3] while a meta-analysis of original and replication results provides an assessment of the evidence pooled across both studies [4].

Perhaps the most commonly used success criterion is the statistical significance of the replication study with an effect estimate in the same direction as in the original study [5–9]. Positive original findings are usually also based on significance, and the procedure is then analogous to the ‘two-trials rule’ in drug development [10, §12.2.8]. The intuition behind this approach is that each study individually should provide sufficient evidence for a claim, which is not guaranteed with other approaches such as meta-analysis or the $Q$ -test. In practice, the standard two-sided significance level 0.05 is often used and so replication success with the two-trials rule is achieved if both one-sided $p$ -values, $p_o$ and $p_r$ , from the original and replication study, respectively, are smaller than $\alpha =0.05/2=0.025$ . The one-sided formulation is convenient to ensure that the effect estimates from the two studies are in the same direction.

The success condition of the two-trials rule $\max \{p_o,p_r\}\le \alpha$ can be rewritten as

$${\displaystyle p_{2\mathrm{T}\mathrm{R}}=max\{p_o,p_r{\}}^2\le {\alpha }^2,}$$ (1.1)

where the combined $p$ -value $p_{2TR}$ of the two-trials rule turns out to be valid [11], i.e. uniformly distributed under the intersection null hypothesis that both the true effect ${\theta }_o$ in the original study and the true effect ${\theta }_r$ in the replication study are null. The two-trials rule therefore controls the overall Type-I error rate at level ${\alpha }^2$ . For $\alpha =0.025$ , the probability to incorrectly declare replication success under the intersection null hypothesis is ${0.025}^2=0.000625$ .

Despite its simplicity, in the replication setting the two-trials rule has important limitations. First of all, the ‘double dichotomization’ at $\alpha =0.025$ can lead to conclusions which seem counterintuitive. For example, the two-trials rule is not fulfilled if $p_o=0.026$ and $p_r=0.001$ (or vice versa), but it is fulfilled when both $p_o$ and $p_r$ are 0.024, although there is less evidence for an effect in the second case [12]. The first example also illustrates that replication success is impossible if the original study just missed statistical significance, no matter how convincing the replication study is. However, some replication projects do interpret non-significant original studies as positive findings and try to replicate them. In the Reproducibility Project: Psychology (RPP) [5], four ’positive’, but non-significant original effects have been included, all with (one-sided) $p$ -values between 0.025 and 0.03. In the Experimental Economics Replication Project (EERP) [6], two ’positive’, but non-significant effects have been included, one with $p_o=0.027$ [13] and one with an even larger $p$ -value, $p_o=0.035$ [14]. The strict application of the two-trials rule would inflate the overall Type-I error rate and reduce trust in the replication finding.

The past and ongoing debate on the use and misuse of $p$ -values [15–17] has led various researchers to advocate for a quantitative interpretation of $p$ -values as measures of the strength of evidence [18] or divergence [19]. This implies that an original study with a relatively large $p$ -value carries only suggestive evidence against the null hypothesis of no effect [20] and requires a more convincing replication result for confirmation than an original study with a smaller $p$ -value. There is in fact empirical evidence that studies with small $p$ -values tend to replicate better than studies with only suggestive evidence. For example, several large-scale replication projects found strong negative Spearman correlations between original $p$ -values and the corresponding (two-sided) replication $p$ -values being smaller than 0.05 [5–7]. Held [21] has shown that more stringent significance levels have improved performance in predicting replication success in an application to the data from the Open Science Collaboration [5] project. The two-trials rule, however, requires the same level of evidence at replication no matter what level of evidence the claim of the original discovery had, as long as it was significant.

A promising alternative is to summarize the total evidence from the two $p$ -values with a combined $p$ -value, but different from the one based on the two-trials rule (1.1). A large number of valid $p$ -value combination methods are available [22,23] and the two-trials rule can serve as a benchmark as it provides the success threshold ${\alpha }^2$ to ensure appropriate control of the overall Type-I error rate [24–26]. Perhaps most prominent is Fisher’s combination method [27], where replication success at level ${\alpha }^2$ is achieved if the product of the $p$ -values $p_o{p}_r\le c_F=\exp \{-0.5{\chi }_4^2(1-{\alpha }^2)\}$ , where ${\chi }_{\nu }^2(\cdot )$ denotes the quantile function of the ${\chi }^2$ -distribution with $\nu$ degrees of freedom. The corresponding combined $p$ -value is

$${\displaystyle p_F=1-Pr\left(X\le -2\log \{p_o{p}_r\}\right),}$$ (1.2)

where $X$ follows a ${\chi }^2$ distribution with 4 d.f., so replication success would be declared if $p_F\le {\alpha }^2$ . However, the goal of replication studies is to confirm an original (positive) finding, but replication success with Fisher’s method can be achieved even if one of the $p$ -values is very large. In fact, replication success at level ${\alpha }^2={0.025}^2$ is guaranteed if $p_o<c_F\approx 0.00006$ , no matter how large $p_r$ is, so Fisher’s method may not even require a replication study to confirm a claim of a new discovery, which is clearly undesirable.

An alternative approach used often in replication projects [5–7,9] is to calculate a combined $p$ -value $p_{\text{MA}}$ with a fixed-effect meta-analysis, also known as the ‘pooled-trials rule’ in drug development [10, §12.2.8]. Let $\mathrm{\Phi }(\cdot )$ denote the standard normal cumulative distribution function and ${\mathrm{\Phi }}^{-1}(\cdot )$ the corresponding quantile function. The meta-analytic $p$ -value $p_{\text{MA}}=1-\mathrm{\Phi }(z_{\text{MA}})$ is identical to the combined $p$ -value from weighted Stouffer’s method [23,28], where

$${\displaystyle z_{\text{MA}}=\frac{{\sigma }_r{z}_o+{\sigma }_o{z}_r}{\sqrt{{\sigma }_o^2+{\sigma }_r^2}}}$$ (1.3)

is the weighted average of the original and replication $z$ -values $z_o={\mathrm{\Phi }}^{-1}(1-p_o)$ and ${\displaystyle z_r={\mathrm{\Phi }}^{-1}(1-p_r)}$ with weights proportional to the standard errors ${\sigma }_r$ and ${\sigma }_o$ of ${\hat{\theta }}_r$ and ${\hat{\theta }}_o$ , respectively. Again, the criterion $p_{\text{MA}}\le {\alpha }^2$ has been used to assess replication success [24,25,29,30], although a threshold larger than 0.000625 (e.g. two-sided 0.005 or even 0.05) is often used in applications. But it is immediate from (1.3) that meta-analysis has similar problems Fisher’s method because $z_{\text{MA}}$ can become large (and hence $p_{\text{MA}}$ small) even if one of the two underlying $z$ -values $z_o$ and $z_r$ is small or even negative (and hence the corresponding $p$ -value large). Recently, Muradchanian et al. [4] have conducted a study to investigate meta-analysis as a replication success metric. They also conclude that meta-analysis is an inappropriate tool if one wants to evaluate whether the replication result is in line with the original result.

In what follows we propose a particular $p$ -value combination method based on the sum of $p$ -values which requires both studies to be convincing, similar to the two-trials rule: Edgington’s method [31] recently rediscovered by Held [32] in the context of drug regulation. For overall Type-I error control at level ${\alpha }^2={0.025}^2$ , success is achieved if the sum of $p$ -values is smaller than ${\displaystyle \sqrt{2}\times 0.025\approx 0.035}$ , so possible if either $p_o$ or $p_r$ is not significant, as long as they are both smaller than $0.035$ . As such, the method can be used in the scenarios encountered in previous replication projects where an original study was borderline non-significant. We also derive a weighted version of Edgington’s method that treats original and replication differently. Giving twice the weight to the replication study results in the necessary (but not sufficient) success conditions $p_o\le 2\alpha$ and $p_r\le \alpha$ . Both versions of Edgington’s method reach opposite conclusions than the two-trials rule for the two examples introduced earlier: replication success at level ${0.025}^2$ is declared if $p_o=0.026$ and $p_r=0.001$ , but not if $p_o$ and $p_r$ are 0.024. The approach is summarized in box 1.

Box 1. Assessing replicability using the sum of p-values.

Overall Type-I error control at level ${\alpha }^2={0.025}^2$ .

Input: one-sided $p$ -values $p_o$ and $p_r$ from original and replication study.

	unweighted	weighted
weights	${\displaystyle w_o=1}$ , ${\displaystyle w_r=1}$	$w_o=1$ , $w_r=2$
replication success criterion	${\displaystyle \begin{array}{rcl}{p}_o+p_r& \le & \sqrt{2}\alpha \\ & \approx & 0.035\end{array}}$	${\displaystyle \begin{array}{rcl}{p}_o+2p_r& \le & 2\alpha \\ & =& 0.05\end{array}}$
significance level for replication study	${\displaystyle \begin{array}{rl}& \sqrt{2}\alpha -p_o\\ \approx & 0.035-p_o\end{array}}$	${\displaystyle \begin{array}{rl}& \alpha -p_o/2\\ =& 0.025-p_o/2\end{array}}$
combined ${\displaystyle p}$ -value	${\displaystyle p_E=(p_o+p_r{)}^2/2}$  (for ${\displaystyle p_o+p_r\le 1}$ )	${\displaystyle p_{E_w}=(p_o+2p_r{)}^2/4}$  (for ${\displaystyle p_o+2p_r\le 1}$ )

Other choices can be made for $\alpha$ and the weights $w_o$ and $w_r$ .

The rest of the article is organized as follows. Edgington’s method is described in §2.1 and extended to include weights in §2.2. In §3, we discuss and compare the conditional Type-I error rate and the project power of the two-trials rule, meta-analysis and Fisher’s and Edgington’s methods. The conditional Type-I error rate is the probability, given the original study result, to incorrectly declare replication success when the true effect is null at replication, while the project power is the probability to correctly declare replication success over both studies in combination. The different methods are applied to the data from four large-scale replication projects in §4 and sample size planning of the replication study is described in §5. Extensions to more than one replication study are described in §6. Finally, some discussion is provided in §7.

2. Additive combination of p-values

2.1. Edgington’s method

Edgington [31] developed a method to combine $p$ -values by adding them. Here, we investigate the use of Edgington’s method in the replication setting, where results from one original and one replication study are available. Under the intersection null hypothesis, the sum of the two $p$ -values:

$${\displaystyle E=p_o+p_r}$$ (2.1)

follows the Irwin–Hall distribution with parameter $n=2$ [33,34], i.e. $E\sim$ IH(2). The cumulative distribution function of the Irwin–Hall distribution can then be used to calculate a valid combined $p$ -value:

$${\displaystyle \begin{array}{l}{p}_E=Pr(\text{IH}(2)\le E)=\{\begin{array}{rl}{E}^2/2& \text{ if }0<E\le 1,\\ -1+2E-E^2/2& \text{ if }1<E\le 2.\end{array}\end{array}}$$ (2.2)

The success condition $p_E\le {\alpha }^2$ can be expressed in terms of the $p$ -value sum $E\le b$ , where the critical value is $b=\sqrt{2}\alpha$ . At the standard $\alpha =0.025$ , the critical value is hence $b\approx 0.035$ . The threshold $b$ can be considered as the available budget for the two $p$ -values, $p_o$ and $p_r$ , and this implies that replication success is possible for a non-significant original (or replication) $p$ -value as long as $p_o$ (or $p_r$ ) $<b=0.035$ . The sum of the $p$ -values for the two examples presented in §1 is $E=0.026+0.001=0.027$ and $E=0.024+0.024=0.048$ , respectively, so replication success is declared in the first example but not the second, in contrast to the two-trials rule but in accordance with intuition. The corresponding combination $p$ -values (2.2) are $p_E=0.0004$ and $p_E=0.001$ , respectively, the first smaller and the second larger than the threshold ${\alpha }^2=0.000625$ .

2.2. Weighted version

An interesting extension of Edgington’s method is to include weights, for example, proportional to the precision of the studies, as in weighted Stouffer’s method. However, in the replication setting one might want to downweight the original and upweight the replication study, for example, with weights $1/3$ and $2/3$ , respectively. This would address concerns that original studies may be subject to questionable research practice and hence prone to bias. Consider the weighted sum of $p$ -values:

$${\displaystyle E_w=w_o{p}_o+w_r{p}_r,}$$ (2.3)

with positive weights $w_o\le w_r$ , then we show in appendix A that the corresponding combination $p$ -value is

$${\displaystyle \begin{array}{l}{p}_{E_w}=\{\begin{array}{rl}\frac{E_w^2}{2w_o{w}_r}& \text{ if }0<E_w\le w_o,\\ \frac{1}{w_r}\left(E_w-\frac{w_o}{2}\right)& \text{ if }{w}_o<E_w\le w_r,\\ 1+\frac{1}{w_o{w}_r}\left(E_w(w_o+w_r)-\frac{(w_o+w_r{)}^2}{2}-\frac{E_w^2}{2}\right)& \text{ if }{w}_r<E_w\le w_o+w_r.\end{array}\end{array}}$$ (2.4)

Note that (2.4) reduces to (2.2) for $w_o=w_r=1$ , and is invariant under multiplication of the weights $(w_o,w_r)$ with a positive constant. The $p$ -value (2.4) therefore only depends on the weight ratio $\stackrel{\mathrm{~}}{w}=w_r/w_o$ of the replication to the original weight. Setting the first line of (2.4) equal to ${\alpha }^2$ gives the available budget $b_w=\sqrt{2w_o{w}_r}\alpha$ for $E_w\le w_o$ .

In the following we will use the weights $w_o=1$ and $w_r=2$ , although other choices can be made, of course. Then

$${\displaystyle \begin{array}{l}{p}_{E_w}=\{\begin{array}{rl}{E}_w^2/4& \text{ if }0<E_w\le 1,\\ E_w/2-1/4& \text{ if }1<E_w\le 2,\\ -\frac{5}{4}+\frac{3}{2}{E}_w-E_w^2/4& \text{ if }2<E_w\le 3.\end{array}\end{array}}$$ (2.5)

For $E_w\le 1$ with small enough $p_o<p_r$ , we have $p_{E_w}=(p_o+2p_r{)}^2/4\approx p_r^2=p_{2TR}$ from (1.1), so weighted Edgington will behave similar to the two-trials rule, whereas the $p$ -value from the unweighted version will then be roughly half as large as $p_{2TR}$ : $p_E=(p_o+p_r{)}^2/2\approx p_r^2/2=p_{2TR}/2$ .

The success condition $p_{E_w}=E_w^2/4\le {\alpha }^2$ can be rewritten as $E_w=p_o+2p_r\le 2\alpha$ , so doubling the replication weight increases the budget from $\sqrt{2}\alpha$ to $2\alpha$ and for $\alpha =0.025$ it will be possible to successfully replicate original studies with $p_o\le 0.05$ . However, the $p$ -value of the replication study now counts twice in $E_w$ , so replication success is impossible if $p_r>\alpha$ , just as with the two-trials rule. For example, the original study by Kuziemko et al. [14] mentioned in §1 had a quite large $p$ -value: $p_o=0.035$ . Conducting a replication would be pointless if analysis is based on the two-trials rule or even unweighted Edgington. However, replication success would be still possible with weighted Edgington, but the replication study has to be quite convincing to achieve success ( $p_r<0.025-0.035/2=0.0075$ ).

3. Operating characteristics

As discussed before, all methods considered control the overall Type-I error rate at level ${\alpha }^2$ . We will now look at two other operating characteristics: conditional Type-I error rate and project power.

3.1. Conditional Type-I error rate

The success condition on the replication $p$ -value $p_r$ with the two-trials rule is always $p_r\le \alpha$ , regardless of the original study result (as long as $p_o\le \alpha$ holds). In contrast, with Edgington’s and Fisher’s methods as well as the meta-analysis criterion the required value of $p_r$ depends on the original $p$ -value $p_o$ . This condition is

$${\displaystyle p_r\le b-p_o}$$ (3.1)

for Edgington’s method,

$${\displaystyle p_r\le (b_w-w_o{p}_o)/w_r}$$ (3.2)

for weighted Edgington’s method,

$${\displaystyle p_r\le min\{c_F/p_o,1\}}$$ (3.3)

for Fisher’s method, and

$${\displaystyle p_r\le 1-\mathrm{\Phi }\left\{\frac{{\mathrm{\Phi }}^{-1}(1-{\alpha }^2)\sqrt{c+1}-z_o}{\sqrt{c}}\right\}}$$ (3.4)

for the meta-analysis criterion, which depends on the variance ratio $c={\sigma }_o^2/{\sigma }_r^2$ . For a very convincing original study (where $p_o\to 0$ , equivalently $z_o\to \infty$ ), the right-hand side in (3.1) tends to $b=\sqrt{2}\alpha$ , in (3.2) to $b_w/w_r=\sqrt{2/\stackrel{\mathrm{~}}{w}}\alpha$ , while the right-hand side in (3.3) and (3.4) converges to $1$ . Replication success can thus be achieved with Fisher’s method and the meta-analysis criterion even if the replication $p$ -value is very large, while this cannot happen with the two-trials rule and Edgington’s method.

Of particular interest in the replication setting is the conditional Type-I error rate, the probability, given the original study result, that the replication study flags success although there is no true effect at replication ( ${\theta }_r=0$ ). For $\alpha =0.025$ , the conditional Type-I error rate is $\alpha =0.025$ , $b-p_o<b=0.035$ and $(b_w-w_o{p}_o)/w_r<b_w/w_r=0.025$ with the two-trials rule and the unweighted and weighted Edgington’s method, respectively, but can become very large with Fisher’s method and the meta-analysis criterion, if $p_o$ is small. For example, suppose the original study had a $p$ -value of $p_o=0.001$ . The conditional Type-I error rate of Edgington’s method at the standard $\alpha =0.025$ level then is 3.4% (unweighted) and 2.45% (weighted). With Fisher’s method and meta-analysis (for $c=1$ ), the conditional Type-I error rate is 5.8% and 7.0%, respectively. Now suppose the original study had a $p$ -value of $p_o=0.0001$ . The conditional Type-I error rate then is 3.53% (unweighted) and 2.495% (weighted) with Edgington’s method, so only slightly larger. However, with Fisher’s method and meta-analysis, the conditional Type-I error rate increases drastically to 58.1% and 19.9%, respectively.

3.2. Project power

The project power is the probability to correctly declare replication success over both studies in combination when both underlying effects are non-null. Most original studies are designed to have 80% power to detect the assumed true effect size at significance level $\alpha$ , but the power can be considerably lower in reality [35,36]. In figure 1, we consider the project power with an original power of $80$ % (figure 1a,b ) and 40% (figure 1c,d ), under the assumption that the true effect sizes are the same ( ${\theta }_r={\theta }_o$ ; figure 1a,c ) or that the true replication effect size is half the true original effect size ( ${\theta }_r={\theta }_o/2$ ; figure 1b,d ). The latter case reflects the shrinkage of effect estimates often encountered in replication projects.

Figure 1.

Project power of the two-trials rule, Edgington’s (both unweighted and weighted) and Fisher’s methods, and the meta-analysis criterion as a function of the relative sample size $c=n_r/n_o$ , assuming that the true effect sizes ${\theta }_o$ and ${\theta }_r$ are the same (a,c), or that the true replication effect size is half the true original effect size ${\theta }_r={\theta }_o/2$ (b,d). The calculations assume that the original study has a power of 80% (a,b) or 40% (c,d) to detect the assumed true effect size at significance level $\alpha$ .

In contrast to the conditional Type-I error rate, the project power of all methods depends on the original power to detect ${\theta }_o$ and the variance ratio $c$ , which can often be interpreted as the relative sample size $c=n_r/n_o$ (replication to original); see appendix B for the derivations. The project power of the two-trials rule cannot become larger than the power in the original study to detect the true effect size (see figure 1). The project power of Edgington’s method is either essentially identical (for small $c$ and the weighted version) or otherwise larger than the project power of the two-trials rule with limit 84% (unweighted) and 87.6% (weighted), respectively, for $c\to \infty$ and an original power of $80$ %. The corresponding values are 46% (unweighted) and 52.5% (weighted) for an original power of $40$ %. The project power of Fisher’s method and the meta-analysis criterion is larger than the project power of both the two-trials rule and Edgington’s method and reaches values close to 100% in the case ${\theta }_r={\theta }_o$ even for a low original power of 40%. However, the price to pay is a considerable increase in conditional Type-I error rate, as discussed in §3.1.

4. Application

The RPP [5], the EERP [6], the Social Sciences Replication Project (SSRP) [7] and the Experimental Philosophy Replicability Project (EPRP) [8] are large-scale replication projects which aimed to replicate important discoveries published in journals from their respective fields. Here, we consider $138$ original studies considered to have a ‘positive’ effect, i.e. either significant or with a ‘trend to significance’ (i.e. non-significant with a $p$ -value just slightly larger than the significance threshold). Several methods were used to assess replication success, such as the two-trials rule, meta-analysis of the original and replication effect estimates or compatibility of the replication effect estimates with a prediction interval based on the original results.

We grouped the 138 positive original studies into significant ( $p_o\le$ threshold) and non-significant ( $p_o>$ threshold) at varying thresholds. The proportion of significant replication studies at the one-sided 0.025 level ( $p_r\le \alpha =0.025$ ) is then calculated for each of the two groups and displayed in figure 2. The proportion of significant replication studies is higher for more convincing original studies (i.e. significant original studies at a smaller significance threshold). This shows that more convincing original studies tend to replicate more than less convincing original studies, and it therefore makes sense to be less stringent with the former as does Edgington’s method, but not the two-trials rule.

Figure 2.

Proportion of significant replication studies ( $p_r\le 0.025$ ) for original studies with $p$ -value below and above the threshold, as a function of the threshold. The points at the bottom represent the original $p$ -values $p_o$ . There are 17 study pairs with original $p$ -value $p_o<0.000001$ that are not shown.

We then calculated the replication rates in the four projects with the two-trials rule ( ${\displaystyle p_{\text{2TR}}\le {\alpha }^2}$ ) and Edgington’s method ( ${\displaystyle p_E\le {\alpha }^2}$ or $p_{E_w}\le {\alpha }^2$ ) for varying levels ${\alpha }^2$ (see figure 3). The replication rates are similar with a tendency of larger success rates with Edgington’s method. This is in line with the larger project power of Edgington’s method discussed in §3.2. For example, at level ${\alpha }^2={0.025}^2$ , the replication rate of the two-trials rule and Edgington’s method are 30.4% versus 31.9% in the RPP, 55.6% versus 61.1% in the EERP, 61.9% for both in the SSRP and 76.7% for both in the EPRP. The conclusions only differ for two study pairs: the original study by Schmidt & Besner [37] and its replication in the RPP, and the original study by Ambrus & Greiner [13] and its replication in the EERP. As the original $p$ -values $p_o=0.028$ and $p_o=0.027$ are slightly larger than $\alpha =0.025$ in both cases, the two-trials rule is not fulfilled. However, as $p_r<0.0001$ and $p_r=0.006$ , respectively, the sum $p_o+p_r$ does not exceed $b=0.035$ and so replication success with Edgington’s method is achieved for both study pairs. Likewise, success is also achieved for the weighted method as $p_o+2p_r\le 2\alpha =0.05$ in both cases.

Figure 3.

Success rate of the RPP, EERP, SSRP and EPRP as a function of the overall Type-I error rate ${\alpha }^2$ with the two-trials rule and Edgington’s method.

We also computed the combined $p$ -values $p_{\text{2TR}}$ , $p_E$ , $p_F$ and $p_{\text{MA}}$ with each of the four methods and plotted them against the replication $p$ -value $p_r$ for non-significant replication studies (see figure 4). By construction, the combined $p$ -value from weighted Edgington’s method is always larger than the success threshold ${\alpha }^2=0.000625$ , and this is also the case for the unweighted version, where the smallest combined $p$ -value $p_E=0.000635$ is just slightly above the threshold. In contrast, with Fisher’s method and the meta-analysis criterion, replication success is often declared although the replication $p$ -value is quite large. There are even three studies with $p_r>0.5$ (so with an effect estimate in the wrong direction) which achieve success at level ${\alpha }^2=0.000625$ with Fisher’s method, and one study with the meta-analysis criterion. This illustrates that Fisher’s method and the meta-analysis criterion are not suited as a replacement for the two-trials rule.

Figure 4.

Combined $p$ -values $p_{\text{2TR}}$ , $p_E$ , $p_{E_w}$ , $p_F$ and $p_{\text{MA}}$ have been calculated from the original and replication $p$ -values $p_o$ and $p_r$ , respectively, for all replication studies considered in the four replication projects. They are plotted against the replication $p$ -value $p_r$ for non-significant replication studies ( $p_r>0.025$ ). Combined $p$ -values in the grey area flag replication success at overall Type-I error rate ${0.025}^2=0.000625$ . The dashed black line is the lower bound $p_r^2$ for $p_{\text{2TR}}$ . The dashed orange line is the lower bound $p_r^2/2$ for $p_E\le 1/2$ . Fisher’s ( $p_F$ ) and meta-analytic ( $p_{\text{MA}}$ ) $p$ -values smaller than 10⁻⁶ are marked with a triangle.

5. Sample size calculation

The sample size of the replication study is usually calculated based on conditional power, i.e. such that the power $1-\beta$ to reach a significant replication effect estimate reaches a certain value, usually 80% or 90%, assuming the original effect estimate is the true one. If, additionally, significance of the original study is required, this corresponds to the sample size calculation based on the two-trials rule. In practice, a standard sample size calculation method is used where the minimal clinically important difference is replaced with the original effect estimate ${\hat{\theta }}_o$ . For example, for a balanced two-sample $z$ -test, the sample size per group in the replication study is calculated as

$${\displaystyle n_r=\frac{2{\tau }^2(z_{1-\alpha }+z_{1-\beta }{)}^2}{{\hat{\theta }}_o^2},}$$ (5.1)

where $\tau$ denotes the common standard deviation of the measurements and $z_{1-u}={\mathrm{\Phi }}^{-1}(1-u)$ denotes the $1-u$ quantile of the standard normal distribution. We note that in some replication projects [7] the original effect estimate ${\hat{\theta }}_o$ in (5.1) is reduced by 25% or even 50% to take into account its possible inflation [38].

It is also possible to express equation (5.1) on the relative scale. In that case, the required relative sample size $c={\sigma }_o^2/{\sigma }_r^2=n_r/n_o$ is calculated as

$${\displaystyle c=\frac{{\left(z_{1-\alpha }+z_{1-\beta }\right)}^2}{z_o^2}.}$$ (5.2)

Sample size calculation based on (5.1) or (5.2) is appropriate if significance of the replication study at level $\alpha$ is the desired criterion for replication success. If instead Edgington’s method will be used, the sample size calculation needs to be appropriately adapted to ensure that the design of the replication study matches the analysis [39]. To do so, the significance level $\alpha$ needs to be replaced with $b-p_o$ in equations (5.1) and (5.2), so now depends on the $p$ -value from the original study. A smaller sample size than with the two-trials rule is therefore required if $b-p_o>\alpha$ , i.e.

$${\displaystyle p_o<b-\alpha =\sqrt{2}\alpha -\alpha =\alpha (\sqrt{2}-1)\approx 0.01.}$$ (5.3)

The weighted version always requires a larger sample size than the two-trials rule, because the required significance level is $\alpha -p_o/2<\alpha$ .

Figure 5 shows the sample size ratio

Figure 5.

Replication sample size ratio of Edgington’s method compared with the two-trials rule to reach 80% (top) and 90% (bottom) power. For conditional power, the sample size ratio is monotonically increasing, while it reaches a minimum at $p_o=0.00009$ for predictive power in the unweighted version. The corresponding sample size reduction is one minus the sample size ratio. The sample size ratio of the weighted version is always monotonically increasing and converges to 1 for $p_o\to 0$ .

$${\displaystyle \frac{(z_{1-b+p_o}+z_{1-\beta }{)}^2}{(z_{1-\alpha }+z_{1-\beta }{)}^2}\text{resp.}\frac{(z_{1-\alpha +p_o/2}+z_{1-\beta }{)}^2}{(z_{1-\alpha }+z_{1-\beta }{)}^2}}$$

of Edgington’s method (unweighted and weighted) versus the two-trials rule for $\alpha =0.025$ , a power of 80% and 90% and $p_o\in [0.00001,0.025]$ . At 80% power, the sample size calculated with unweighted Edgington’s method can be up to 10.6% smaller than with the two-trials rule. At 90% power, the sample size reduction looks very similar with a maximum of 9.2%. However, if $p_o>0.01$ , the required sample size with Edgington’s method is larger than with the two-trials rule. The weighted version always requires a larger sample size, but smaller than the unweighted version if $p_o$ is close to $\alpha =0.025$ .

A drawback of conditional power is that it does not take the uncertainty of the original result into account and hence can lead to underpowered replication studies. One way to take into account the uncertainty of the original result is to use ‘predictive power’ instead [40]. The relative sample size based on predictive power is generally larger than based on conditional power. The sample size reduction of Edgington’s method compared with the two-trials rule can be even more pronounced and reaches a value of 11.2% (10.3%) at $p_o=0.00009$ ( $p_o=0.0002$ ) for 80% (90%) predictive rather than conditional power (see figure 5).

6. Extensions to more than one replication study

It has been argued that a single replication study will often not be sufficient and that more than one replication study is needed to provide an unambiguous evaluation of replicability [41]. Edgington’s method can also be used if more than one replication study is conducted. A simple approach would be to combine the replication $p$ -values into an overall replication $p$ -value and then use Edgington’s method for one original and one replication $p$ -value, as introduced in this article. However, Edgington’s method can also be applied directly to the individual $p$ -values, as we now illustrate in the case of three studies (one original and two replications). Now the sum $E_3=p_o+p_{r1}+p_{r2}$ of the three $p$ -values needs to be smaller than the new budget $b_3=0.16$ to ensure overall Type-I error control at level ${0.025}^2$ [32]. An interesting aspect of this approach is that it can be used to save resources if the replication studies are conducted sequentially. Indeed, there will be no point in conducting the second replication study if the sum of $p$ -values $E_2=p_o+p_{r1}$ from the original and the first replication study is already larger than $b_3$ . Otherwise, a second replication study at significance level $b_3-E_2$ can be planned and we would flag replication success if $E_3=E_2+p_{r2}\le b_3$ holds. Such a sequential conduct of replication studies has been suggested by Hedges & Schauer [41, p. 567] because ‘a single initial replication may be one effort in a sequence of replications, and as researchers conduct additional subsequent replications, eventually a preponderance of evidence will support more sensitive analyses’.

A refined version of this approach has been proposed by Held [32, §4], allowing to stop for success already after the first replication study. The approach is based on so-called alpha-spending [42], distributing the overall Type-I error rate ${\alpha }^2$ to the analysis after the first and the second replication study. Alpha-spending is a method originally proposed for interim analyses in clinical trials. Specifically, Fisher’s method has been proposed for the evaluation of experiments with an adaptive interim analysis based on the $p$ -values of two subsamples of the study [43]. Closer to our approach is the method by Chang [44] who derives stopping rules based on the sum of the $p$ -values for each subsample of the trial.

The application of the alpha-spending approach to the replication setting is illustrated in figure 6, which shows the budget $b_2$ for $E_2$ and $b_3$ for $E_3$ depending on the proportion of ${\alpha }^2$ that is spent on the analysis after the first replication. For example, if we spend half of ${\alpha }^2$ on the first replication, we can stop for replication success if $E_2=p_o+p_{r1}<b_2=0.025$ holds. If this is not the case but at least $E_2<b_3=0.13$ holds, we would plan and conduct a second replication study at significance level $b_3-E_2$ . If eventually $E_3=E_2+p_{r2}\le b_3=0.13$ holds, we have achieved success after the second replication study. The combined procedure thus not only allows to stop for replication success or failure after the first replication study but offers a third possibility to conduct a second replication study if $b_2=0.025<E_2<b_3=0.13$ . The approach could also be extended to more than two replication studies and weights could also be introduced.

Figure 6.

Budget $b_2$ for $E_2=p_o+p_{r1}$ and $b_3$ for $E_3=p_o+p_{r1}+p_{r2}$ , depending on the proportion of ${\alpha }^2={0.025}^2$ spent on the first replication study. The points denote the available budget if half of ${\alpha }^2$ is spent after the first replication.

7. Discussion

We propose to use the sum of the $p$ -values, also known as Edgington’s method, instead of the two-trials rule in the assessment of replication success. An unweighted and a weighted version are considered. In cases where it can be safely assumed that the original study follows the same standards as the replication study [45], we recommend to use the unweighted version. In cases where the original study may be subject to questionable research practices or publication bias, we recommend to give more weight to the replication study. The exact choice of the weight depends on how much we distrust the original study result. Both the unweighted and the weighted methods exactly control the overall Type-I error rate at level ${\alpha }^2$ and have an acceptable bound on the conditional Type-I error rate, namely, $b=0.035$ and $\alpha =0.025$ , respectively. These numbers are for the conventional (but arbitrary) $\alpha =0.025$ and a weight ratio of 2, and in principle, other values could be used.

The success bound for the replication $p$ -value $p_r$ with Edgington’s method is not fixed at $\alpha =0.025$ but varies between $0$ and $b=0.035$ (unweighted) or between $0$ and $\alpha =0.025$ (weighted), depending on the original $p$ -value $p_o$ . Replication success is possible for original studies that missed traditional statistical significance, as long as $p_o\le 0.035$ (unweighted) or $p_o\le 2\alpha =0.05$ (weighted). While these bounds are less stringent than with the two-trials rule, they are also different from the more elaborate sceptical $p$ -value [46]. The sceptical $p$ -value has been developed specifically for replication studies and depends on the two $p$ -values, $p_o$ and $p_r$ , but also on the variance ratio, so treats original and replication studies not as exchangeable. The controlled sceptical $p$ -value [47] ensures exact overall Type-I error rate control, just as all methods discussed in this article. It also allows for replication success if the original study is non-significant and can be used for sample size calculations. However, the method is more complicated and therefore more difficult to communicate. Edgington’s method can be seen as a simple compromise between the two-trials rule and the controlled sceptical $p$ -value, valuing the combined evidence from both studies while ensuring that both studies support the alternative hypothesis. Of course, researchers may still want to quantify other aspects of replicability, such as statistical consistency of original and replication effect estimates, for which other methods, such as the $Q$ -test, could be used [41]. In future work, we plan to conduct a simulation study to compare the operating characteristics of Edgington’s method with the sceptical $p$ -value and alternative methods in the presence of publication bias and other questionable research practices [30,48].

One advantage of Edgington’s method is that it can be easily applied to non-normal or non-standard settings, for example, based on the $t$ -test, a comparison of proportions or the log-rank test for survival data. For example, for a sample size calculation based on the $t$ -test, the R function power.t.test() can be used to calculate the required replication sample size $n_r$ . The argument delta needs to be set to the original effect estimate ${\hat{\theta }}_o$ (perhaps incorporating some additional shrinkage) and the argument sig.level to $\sqrt{2}\alpha -p_o$ (Edgington) or $\alpha -p_o/2$ (weighted Edgington) rather than $\alpha$ (two-trials rule).

Appendix A. Weighted sum of p-values

The weighted sum of $p$ -values (2.3) with weights $w_o\le w_r$ can be written as

$${\displaystyle E_w=q_o+q_r,}$$

where $q_o\sim \mathrm{U}(0,w_o)$ and $q_r\sim \mathrm{U}(0,w_r)$ are independent uniform under the intersection null hypothesis. The density function of $E_w$ can be computed as the convolution of the densities of $q_o$ and $q_r$ and follows a trapezoidal distribution [51]:

$${\displaystyle f(e_w)=\{\begin{array}{rl}\frac{e_w}{w_o{w}_r}& \text{ if }0<e_w\le w_o,\\ \frac{1}{w_r}& \text{ if }{w}_o<e_w\le w_r,\\ \frac{w_o+w_r-e_w}{w_o{w}_r}& \text{ if }{w}_r<e_w\le w_o+w_r.\end{array}}$$

The cumulative distribution function of $E_w$ is therefore

$${\displaystyle F(e_w)=\{\begin{array}{rl}\frac{e_w^2}{2w_o{w}_r}& \text{ if }0<e_w\le w_o,\\ \frac{1}{w_r}\left(e_w-\frac{w_o}{2}\right)& \text{ if }{w}_o<e_w\le w_r,\\ 1+\frac{1}{w_o{w}_r}\left(e_w(w_o+w_r)-\frac{(w_o+w_r{)}^2}{2}-\frac{e_w^2}{2}\right)& \text{ if }{w}_r<e_w\le w_o+w_r.\end{array}}$$

A valid combined $p$ -value is obtained by plugging $E_w$ into the cumulative distribution function: $p_{E_w}=F(E_w)$ .

Appendix B. Project power

First assume that both effects are the same, so ${\theta }_r={\theta }_o$ . As a result, ${\displaystyle z_o\sim \mathrm{N}(\mu ,1)}$ and ${\displaystyle z_r\sim N(\mu \sqrt{c},1)}$ . The project power of the two-trials rule can be found in Held et al. [52, §3.3]. The project power of all other methods is calculated as

$${\displaystyle \int Pr(\text{Replication success}\mid z_o)\varphi (z_o-\mu )\mathrm{d}{z}_o.}$$

Specifically, Edgington’s method has project power

$$\begin{gathered} {\displaystyle {\int }_{{\mathrm{\Phi }}^{-1}(1-b)}^{\mathrm{\infty }}Pr(p_r\le b-p_o)\varphi (z_o-\mu )\mathrm{d}{z}_o \\ {\int }_{{\mathrm{\Phi }}^{-1}(1-b)}^{\mathrm{\infty }}Pr(z_r\ge {\mathrm{\Phi }}^{-1}(1-b+p_o))\varphi (z_o-\mu )\mathrm{d}{z}_o \\ {\int }_{{\mathrm{\Phi }}^{-1}(1-b)}^{\mathrm{\infty }}Pr(z_r\ge {\mathrm{\Phi }}^{-1}(2-b-\mathrm{\Phi }(z_o)))\varphi (z_o-\mu )\mathrm{d}{z}_o \\ ={\int }_{{\mathrm{\Phi }}^{-1}(1-b)}^{\mathrm{\infty }}\left[1-\mathrm{\Phi }\left\{{\mathrm{\Phi }}^{-1}\left(2-b-\mathrm{\Phi }(z_o)\right)-\mu \sqrt{c}\right\}\right]\varphi (z_o-\mu )\mathrm{d}{z}_o.} \end{gathered}$$ (B 1)

The project power of weighted Edgington’s method with weight ratio $\stackrel{\mathrm{~}}{w}=w_r/w_o$ turns out to be

$${\displaystyle {\int }_{{\mathrm{\Phi }}^{-1}(1-b_w/w_o)}^{\mathrm{\infty }}\left[1-\mathrm{\Phi }\left\{{\mathrm{\Phi }}^{-1}\left(1+1/\tilde{w}-\sqrt{2/\tilde{w}}\alpha -\mathrm{\Phi }(z_o)/\tilde{w}\right)-\mu \sqrt{c}\right\}\right]\varphi (z_o-\mu )\mathrm{d}{z}_o.}$$ (B 2)

The project power of Fisher’s method is

$${\displaystyle \begin{array}{l}{\int }_0^{\mathrm{\infty }}f(z_o)\varphi (z_o-\mu )\mathrm{d}{z}_o\text{ with }f(z_o)=\{\begin{array}{ll}1-\mathrm{\Phi }\left\{{\mathrm{\Phi }}^{-1}\left(1-\frac{c_F}{1-\mathrm{\Phi }(z_o)}\right)-\mu \sqrt{c}\right\},& \text{if }1-\mathrm{\Phi }(z_o)\ge c_F\\ 1,& \text{otherwise},\end{array}\end{array}}$$ (B 3)

and the project power of the meta-analysis criterion is

$${\displaystyle {\int }_0^{\mathrm{\infty }}\left[1-\mathrm{\Phi }\left\{\frac{{\mathrm{\Phi }}^{-1}(1-{\alpha }^2)\sqrt{c+1}-z_o}{\sqrt{c}}\right\}-\mu \sqrt{c}\right]\varphi (z_o-\mu )\mathrm{d}{z}_o.}$$ (B 4)

The project power of Edington’s method converges to

$${\displaystyle 1-\mathrm{\Phi }\left({\mathrm{\Phi }}^{-1}(1-b)-{\mathrm{\Phi }}^{-1}(1-\alpha )-{\mathrm{\Phi }}^{-1}(1-\beta )\right)}$$

for $c\to \infty$ , while the corresponding limit is

$${\displaystyle 1-\mathrm{\Phi }\left({\mathrm{\Phi }}^{-1}(1-b_w/w_o)-{\mathrm{\Phi }}^{-1}(1-\alpha )-{\mathrm{\Phi }}^{-1}(1-\beta )\right)}$$

for weighted Edgington’s method and 1 for the other two methods.

If ${\theta }_r={\theta }_o/2$ , the term $\mu \sqrt{c}$ in (B 1), (B 2), (B 3) and (B 4) needs to be divided by $2$ .

Ethics

This work did not require ethical approval from a human subject or animal welfare committee.

Data accessibility

The R package Replication Success available on CRAN at: https://CRAN.R-project.org/package=ReplicationSuccess has been used for the sample size calculations. The data of the RPP, EERP, SSRP and EPRP are available through the command data("RProjects"). All p-values have been recalculated based on Fisher’s z-transformation as described in [49], electronic supplementary material]; see also help("RProjects"). The code to reproduce the calculations in this paper is available at [50]. Functions to compute Edgington’s combined p-value (pEdgington) and associated power (powerEdgington) and sample size calculations (sampleSizeEdgington) are available in the development version of the ReplicationSuccess package which can be installed with remotes::install_github(repo = "crsuzh/ReplicationSuccess", ref = "edgington") (requires the remotes package available on CRAN).

Declaration of AI use

We have not used AI-assisted technologies in creating this article.

Authors’ contributions

L.H.: conceptualization, formal analysis, methodology, project administration, software, visualization, writing—original draft, writing—review and editing; S.P.: software, visualization, writing—review and editing; C.M.: formal analysis, methodology, software, visualization, writing—original draft, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

No funding has been received for this article.