Reply to Pek et al.: Science is not the signal detection problem it is ordinarily thought to be

In our target article (1), we note that signal detection theory evolved (2) from Fisher’s (3) null hypothesis significance testing (NHST) approach (H₀: δ = 0, H₁: δ ≠ 0) as elaborated by Neyman and Pearson (4), who additionally proposed considering the specific magnitude of the alternative hypothesis (H₀: δ = 0, H₁: δ = μ). We then argue that this perspective is a useful fiction, not an accurate depiction of underlying reality. The true, underlying effect size (δ) associated with an experimental protocol is more reasonably conceptualized as having been drawn from a continuous distribution, and it is never equal to zero.

Pek et al. (5) argue that the Fisherian perspective can be revised by dropping the null hypothesis fiction that δ = 0. We certainly agree. Following Jones and Tukey (6), they further argue that a modified goal of NHST might be to detect directionally correct effects. This perspective is based on an accurate binary depiction of underlying reality, namely, that δ is in one direction (δ > 0) or the other (δ < 0). Given that nonfictional premise, science can be properly construed as a signal detection problem.

This argument seems perfectly sound to us, and it is entirely consistent with our own view of original science. As we put it: “A P < 0.05 finding should not be regarded as a scientifically established discovery; instead, it should be regarded as a provisional finding, one that is likely in the right direction but with an observed effect size that is inflated to an unknown degree” (ref. 1, p. 5561). It is provisional because the underlying effect size might be too close to 0 to matter (hence the need for large-N direct replications of important findings). Unfortunately, Pek et al. (5) do not address our vision of science at all. Instead, they simply argue that it is possible to imagine an unconventional version of Fisherian NHST that views science as a signal detection problem. It is a fair point, but no one conducts science from that perspective. Our point is that science is not the signal detection problem it is ordinarily thought to be (Fig. 1). Even so, this account is a useful fiction for original science.

Fig. 1.

The standard Fisherian view of NHST science, as elaborated by Neyman and Pearson (H₀: δ = 0, H₁: δ = μ) (4). Many scientists use this signal detection model to compute statistical power and to assess statistical significance. For such purposes, we argue that the model serves as a useful fiction. However, others assume (mistakenly in our view) that this model provides an accurate depiction of underlying reality. They then use it to justify proposed reforms to science, such as encouraging large-N experiments for original research (e.g., ref. 9). Such reforms, we argue, run the risk of making science worse, not better.

Pek et al. (5) also argue that our complex simulations of science were pointless because regression to the mean associated with using a P < 0.05 filter can be demonstrated in a much more parsimonious way. However, our simulations were not conducted to demonstrate inevitable regression to the mean, a statistical phenomenon that we discussed well before (and without reference to) our simulations. Indeed, others called attention to this issue long before we did (e.g., ref. 7). Our simulations were conducted 1) to estimate regression to the mean in the replication results reported by OSC2015 (8) and 2) to illustrate the problem associated with large-N studies in original science. Pek et al. (5) do not address the actual purpose of our simulations. More to the point, they do not engage our vision of science at all.