Reproducibility in science: important or incremental?

In 2016, a survey¹ was published in Nature in which more than 50% of researchers agreed that there was a substantial reproducibility crisis in science. Nearly half of the researchers surveyed cited the pressure to publish as a major contributor to this crisis.¹ Publishing studies of reproducibility has been notoriously difficult, as highlighted by our own recent experience.

In 2019, we submitted a manuscript describing using a new genomic approach to investigate a previously studied tuberculosis outbreak in northern Canada. In doing so, we found a superspreading event that we had not detected in the original analysis, which was linked to specific locations in the community and potentially led to 17 secondary cases (making up 34% of the entire outbreak, in contrast to four secondary cases as previously thought). Although the local public health unit found this information useful, given that transmission is ongoing in this region, the reviewer and editorial comments were decidedly less enthusiastic. Even though it is possible and indeed reasonable for people to disagree with the importance of one's findings, we were struck by this statement from a reviewer: “previously published epidemiological results are of weak epidemiological interest”, followed by an editorial declaration of “incremental benefit”. Statements like these are inherently subjective and are at the heart of the reproducibility crisis.

Reproducibility studies are crucial to the advancement of science. We would suggest that they are particularly important in a field like genomic epidemiology, which is relatively new and where the methods (both laboratory and analytical) are rapidly evolving. It is important to recognise that, as newer methods are developed for this field, these might offer greater resolution or accuracy than those used in the past—as did next-generation sequencing compared with classical genotyping. In genomic epidemiology, no gold standard currently exists for analysis; bioinformatics pipelines for the analysis of these data are generally not standardised across groups, and new tools continue to be developed or refined. In an important step towards reproducibility, several research groups2, 3, 4 are investigating the implications of the lack of standardisation, to assess how differences might affect epidemiological inferences or antimicrobial resistance predictions.

In addition to these efforts, we propose that an important part of reproducibility in genomic epidemiology is to periodically revisit and update previous analyses. Although we all strive to ensure that results are correct by using the best analytical approaches available at the time, as methods change, a logical consequence is that our results and subsequent inferences might change too. Such changes can have important implications not only for research, but also for public health practice, such as changing our understanding of an outbreak, transmission networks, or people who are at risk.

Our study has since been published in eLife,⁵ where it received thoughtful and constructive reviews that improved the quality of the paper, but many authors of reproducibility studies have not been so fortunate. According to the 2016 Nature survey,¹ only 24% of researchers who had failed to reproduce another group's study actually tried to publish a reproducibility study, and of those who tried to publish, only 68·5% succeeded in doing so.¹ In our opinion, journals should be encouraging and supporting the publication of reproducibility studies, rather than casting them aside as merely incremental or lacking novelty. The push for novelty above all else has helped facilitate this crisis, by discouraging researchers from revisiting their own results and those of others in favour of pursing new, arguably more publishable studies.

The value of reproducibility studies has become even clearer in the current confusion around estimating the true seroprevalence of SARS-CoV-2, incidence of infection, and associated mortality. These are real-time examples showing the importance of refining the methods that we use for epidemiology and of carefully scrutinising our own previous work and that of others. Although Mycobacterium tuberculosis has been causing disease for considerably longer than SARS-CoV-2, much remains to be learned about both these pathogens and the methods we use to study them.