Opinion: Is science really facing a reproducibility crisis, and do we need it to?

Abstract

Efforts to improve the reproducibility and integrity of science are typically justified by a narrative of crisis, according to which most published results are unreliable due to growing problems with research and publication practices. This article provides an overview of recent evidence suggesting that this narrative is mistaken, and argues that a narrative of epochal changes and empowerment of scientists would be more accurate, inspiring, and compelling.

Is there a reproducibility crisis in science? Many seem to believe so. In a recent survey by the journal Nature, for example, around 90% of respondents agreed that there is a “slight” or “significant” crisis, and between 40% and 70% agreed that selective reporting, fraud, and pressures to publish “always” or “often” contribute to irreproducible research (1). Results of this nonrandomized survey may not accurately represent the population of practicing scientists, but they echo beliefs expressed by a rapidly growing scientific literature, which uncritically endorses a new “crisis narrative” about science (an illustrative sample of this literature is shown in Fig. 1 and listed in Dataset S1).

Fig. 1.

Number of Web of Science records that in the title, abstract, or keywords contain one of the following phrases: “reproducibility crisis,” “scientific crisis,” “science in crisis,” “crisis in science,” “replication crisis,” “replicability crisis.” Records were classified by the author according to whether, based on title and abstracts, they implicitly or explicitly endorsed the crisis narrative described in the text (red), or alternatively questioned the existence of such a crisis (blue), or discussed “scientific crises” of other kinds or could not be classified due to insufficient information (gray). The complete dataset, which includes all titles and abstracts and dates back to the year 1933, is available in Dataset S1. This sample is merely illustrative, and does not include the numerous recent research articles and opinion articles that discuss the “science is in crisis” narrative without including any of the above sentences in the title, abstract, or keywords.

Put simply, this new “science in crisis” narrative postulates that a large and growing proportion of studies published across disciplines are unreliable due to the declining quality and integrity of research and publication practices, largely because of growing pressures to publish and other ills affecting the contemporary scientific profession.

I argue that this crisis narrative is at least partially misguided. Recent evidence from metaresearch studies suggests that issues with research integrity and reproducibility, while certainly important phenomena that need to be addressed, are: (i) not distorting the majority of the literature, in science as a whole as well as within any given discipline; (ii) heterogeneously distributed across subfields in any given area, which suggests that generalizations are unjustified; and (iii) not growing, as the crisis narrative would presuppose. Alternative narratives, therefore, might represent a better fit for empirical data as well as for the reproducibility agenda.

How Common Are Fabricated, False, Biased, and Irreproducible Findings?

Scientific misconduct and questionable research practices (QRP) occur at frequencies that, while nonnegligible, are relatively small and therefore unlikely to have a major impact on the literature. In anonymous surveys, on average 1–2% of scientists admit to having fabricated or falsified data at least once (2). Much higher percentages admit to other QRP, such as dropping data points based on a gut feeling or failing to publish a contradictory result. However, the percentage of scientific literature that is actually affected by these practices is unknown, and evidence suggests that it is likely to be smaller, at least five times smaller according to a survey among psychologists (3). Data that directly estimate the prevalence of misconduct are scarce but appear to corroborate this conclusion. Random laboratory audits in cancer clinical trials, for example, found that only 0.28% contained “scientific improprieties” (4), and those conducted among Food and Drug Administration clinical trials between 1977 and 1988 found problems sufficient to initiate “for cause” investigations only in 4% of cases (5). Visual inspections of microbiology papers suggested that between 1% and 2% of papers had been manipulated in ways that suggested intentional fabrication (6, 7).

The occurrence of questionable or flawed research and publication practices may be revealed by a high rate of false-positives and “P-hacked” (8) results. However, while these issues do appear to be more common than outright scientific misconduct, their impact on the reliability of the literature appears to be contained. Analyses based on the distribution of P values reported in the medical literature, for example, suggested a false-discovery rate of only 14% (9). A similar but broader analysis concluded that P-hacking was common in many disciplines and yet had minor effects in distorting conclusions of meta-analyses (10). Moreover, the same analysis found a much stronger “evidential value” in the literature of all disciplines, which suggests that the majority of published studies are measuring true effects, a finding that again contradicts the belief that most published findings are false-positives. Methodological criticisms suggest that these and similar studies may be underestimating the true impact of P-hacking (11, 12). However, to the best of my knowledge, there is no alternative analysis that suggests that P-hacking is severely distorting the scientific literature.

Low statistical power might increase the risk of false-positives (as well as false-negatives). In several disciplines, the average statistical power of studies was found to be significantly below the recommended 80% level (13–16). However, such studies showed that power varies widely between subfields or methodologies, which should warn against making simplistic generalizations to entire disciplines (13–16). Moreover, the extent to which low power generates false-positives depends on assumptions about the magnitude of the true underlying effect size, level of research bias, and prior probabilities that the tested hypothesis is in fact correct (17). These assumptions, just like statistical power itself, are likely to vary substantially across subfields and are very difficult to measure in practice. For most published research findings to be false in psychology and neuroscience, for example, one must assume that the hypotheses tested in these disciplines are correct much less than 50% of the time (14, 18). This assumption is, in my opinion, unrealistic. It might reflect the condition of early exploratory studies that are conducted in a theoretical and empirical vacuum, but not that of most ordinary research, which builds upon previous theory and evidence and therefore aims at relatively predictable findings.

It may be counter-argued that the background literature that produces theory and evidence on which new studies are based is distorted by publication and other reporting biases. However, the extent to which this is the case is, again, likely to vary by research subfield. Indeed, in a meta-assessment of bias across all disciplines, small-study effects and gray-literature bias (both possible symptoms of reporting biases) were highly heterogeneously distributed (19). This finding was consistent with evidence that studies on publication bias may themselves be subject to a publication bias (20), which entails that fields that do not suffer from bias are underrepresented in the metaresearch literature.

The case that most publications are nonreproducible would be supported by meta-meta-analyses, if these had shown that on average there is a strong “decline effect,” in which initially strong “promising” results are contradicted by later studies. While a decline effect was measurable across many meta-analyses, it is far from ubiquitous (19). This suggests that in many meta-analyses, initial findings are refuted, whereas in others they are confirmed. Isn’t this what should happen when science is functional?

Ultimately, the debate over the existence of a reproducibility crisis should have been closed by recent large-scale assessments of reproducibility. Their results, however, are either reassuring or inconclusive. A “Many labs” project reported that 10 of 13 studies taken from the psychological literature had been consistently replicated multiple times across different settings (21), whereas an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22). The largest reproducibility initiative to date suggested that in psychological science, reproducibility was below 50% (23). This latter estimate, however, is likely to be too pessimistic for at least two reasons. First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23–25). Second, and more importantly, because how reproducibility ought to be measured is the subject of a growing methodological and philosophical debate, and reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27). Indeed, the very notion of “reproducible research” can be confusing, because its meaning and implications depend on what aspect of research is being examined: the reproducibility of research methods can in principle be expected to be 100%; but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies, for reasons that have nothing to do with questionable research and publication practices (28).

Are These Problems Getting Worse?

In light of multiple recent studies, there is no evidence that scientific misconduct and QRPs have increased. The number of yearly findings of scientific misconduct by the US Office of Research Integrity (ORI) has not increased, nor has the proportion, of all ORI investigations, that resulted in a finding of misconduct (29). Retractions have risen sharply in absolute terms, but the number of retractions per retracting journals has not, suggesting that the trend is due to the diffusion and improvement of journal retraction policies and practices (29). Errata and corrections have also not increased, nor has the rate of statistical errors made in mainstream psychological journals (29, 30).

The questionable practice known as “salami-slicing,” in which results are fractionalized to increase publication output, is widely believed to be on the rise. However, there is no evidence that scientists are publishing more papers today than in the 1950s, once coauthorship is adjusted for (31). Indeed, assessments in various disciplines suggest that, far from becoming increasingly short and trivial, published studies are getting longer, more complex, and richer in data (e.g., refs. 32–34).

Biases in research and reporting were suggested to be on the rise by multiple independent studies, which had found that the relative proportion of “positive” and “statistically significant” results reported in article abstracts has increased over the years (35–37). However, the aforementioned evidence that papers in many (and maybe most) disciplines are becoming longer and more complex suggests that negative results may not be disappearing from the literature, as originally suggested, but perhaps only from abstracts. Negative results, in other words, may be increasingly embedded in longer publications that contain multiple results, and they therefore remain accessible to any researcher interested in finding them.

Finally, pressures to publish have not been convincingly linked to evidence of bias or misconduct. Earlier studies that compared the scientific productivity of countries offered some support for such a link (38, 39). However, later, finer-grained analyses offered contrary evidence, by showing that researchers that publish at higher frequency, in journals with higher impact factor, and in countries where pressures to publish are high, are equally or more likely to correct their work, less likely to publish papers that are retracted, less likely to author papers that contain duplicated images, and less likely to author papers reporting overestimated effects (19, 40, 41). The risk of misconduct and QRPs appears to be higher among researchers in countries that are increasingly represented in the global scientific literature, like China or India (7, 40). Global demographic changes, therefore, might contribute to a rise in the proportion of papers affected by scientific misconduct, but such a trend would have little to do with rising pressures to publish in Western countries.

Do We Need the “Science in Crisis” Narrative to Promote Better Science?

To summarize, an expanding metaresearch literature suggests that science—while undoubtedly facing old and new challenges—cannot be said to be undergoing a “reproducibility crisis,” at least not in the sense that it is no longer reliable due to a pervasive and growing problem with findings that are fabricated, falsified, biased, underpowered, selected, and irreproducible. While these problems certainly exist and need to be tackled, evidence does not suggest that they undermine the scientific enterprise as a whole. Science always was and always will be a struggle to produce knowledge for the benefit of all of humanity against the cognitive and moral limitations of individual human beings, including the limitations of scientists themselves.

The new “science is in crisis” narrative is not only empirically unsupported, but also quite obviously counterproductive. Instead of inspiring younger generations to do more and better science, it might foster in them cynicism and indifference. Instead of inviting greater respect for and investment in research, it risks discrediting the value of evidence and feeding antiscientific agendas.

Furthermore, this narrative is not actually new. Complaints about a decline in the quality of research recur throughout the history of science, right from its beginnings (42, 43). Only two elements of novelty characterize the current “crisis.” The first is that the validity of these concerns is being assessed scientifically by a global metaresearch program, with results that have been briefly overviewed above (44). The second element of historical novelty is the rising power of information and communication technologies, which are transforming scientific practices in all fields, just as they are transforming all other aspects of human life. These technologies promise to make research more accurate, powerful, open, democratic, transparent, and self-correcting than ever before. At the same time, this technological revolution creates new expectations and new challenges that metaresearchers are striving to address.

Therefore, contemporary science could be more accurately portrayed as facing “new opportunities and challenges” or even a “revolution” (45). Efforts to promote transparency and reproducibility would find complete justification in such a narrative of transformation and empowerment, a narrative that is not only more compelling and inspiring than that of a crisis, but also better supported by evidence.