Reproducibility and Research Integrity

Reproducibility—the ability of independent researchers to obtain the same (or similar) results when repeating an experiment or test—is one of the hallmarks of good science (Popper 1959). Reproducibility provides scientists with evidence that research results are objective and reliable and not due to bias or chance (Rooney et al 2016). Irreproducibility, by contrast, may indicate a problem with any of the steps involved in the research such as, but not limited to, the experimental design, variability of biological materials (such as cells, tissues or animal or human subjects), data quality or integrity, statistical analysis, or study description (Landis et al 2012, Shamoo and Resnik 2015). Although researchers have understood the importance of reproducibility for quite some time (Shamoo and Annau 1987), in the last few years the issue has become a pressing concern because of an increasing awareness among scientists and the public that the results of many studies published in peer-reviewed journals are not reproducible (Iaonnidis 2005, The Economist 2013, Collins and Tabak 2014, McNutt 2014, Baker 2015).

Some of the irreproducibility in scientific research may be due to data fabrication or falsification (Shamoo 2013, 2016; Collins and Tabak 2014, Kornfeld and Titus 2016). Misconduct or suspected misconduct accounts for more than two-thirds of retractions (Fang et al 2012). The website Retraction Watch (2016) keeps track of papers which have been retracted due to misconduct or other problems. Approximately 2% of scientists claim that they have fabricated or falsified data at some point in their careers (Fanelli 2009). This percentage may underestimate the actual rate of misconduct because respondents may be unwilling to admit to engaging in illegal or unethical behavior, even in anonymous surveys. Even if the rate of misconduct is low, it still represents a serious ethical problem that can undermine the reproducibility, integrity and trustworthiness of research (Shamoo and Resnik, 2015).

For example, Gottmann et al (2001) compared 121 rodent carcinogenicity assays from the National Cancer Institute/National Toxicology Program and the Carcinogenic Potency Database and found that reproducibility was only 57%. To explain the declining success rates in Phase II drug trials documented by Arrowsmith (2011), scientists from a large pharmaceutical company speculated that poor quality pre-clinical research may be at fault. To test their hypothesis, they analyzed 67 in-house drug target validation projects and found that only 20–25% were reproducible (Prinz et al 2012). In response to evidence of irreproducible results in psychology, a group of 270 researchers attempted to reproduce 100 experiments published in three top psychology journals in 2008. They found that the percentage of studies reporting statistically significant results declined from 97% for the original studies to 36% for the replications, and effects sizes decreased by 50% (Open Science Collaboration 2015). Irreproducibility of behavioral studies may be due, in part, to variations in populations and difficulties with accurately measuring behavioral parameters (Shamoo 2016). While most of the attention has focused on irreproducibility in biomedical (Kilkenny et al 2009, Landis et al 2012, Pusztai et al 2013, Begley and Ioannidis 2015) and psychological research (Open Science Collaboration 2015), many are concerned that reproducibility problems may also plague “hard” sciences like physics and chemistry (The Economist 2013, Nature 2016).

Scientific journals, funding agencies, and researchers have responded to the reproducibility “crisis” by articulating standards for designing experiments, analyzing data, and reporting methods, materials, data, and results (Landis et al 2012, Pusztai et al 2013, Collins and Tabak 2014, McNutt 2014, The Science Exchange Network 2014, Nature 2014a, National Institutes of Health 2016, Rooney et al 2016). While many of these standards tend to be discipline-specific, some apply across disciplines. For example, statistical power analysis can help to ensure that one’s sample is large enough to detect a significant effect in biomedical, physicochemical, or behavioral research; randomization and blinding can control for bias in clinical trials or animal experiments; and auditing of data and other research records can help reduce errors and inconsistencies in many fields of science (Shamoo and Resnik 2015, National Institutes of Health 2016, Shamoo 2016).

Reproducibility is not just a scientific issue; it is also an ethical one. When scientists cannot reproduce a research result, they may suspect data fabrication or falsification. In several well-known cases, reproducibility issues led to allegations of data fabrication or falsification. For example, in 1986, postdoctoral researcher Margot O’Toole accused her supervisor, Tufts University pathology assistant professor Thereza Imanishi-Kari, of fabricating and falsifying data in a National Institutes of Health (NIH)-funded study on using foreign genes to stimulate antibody production in mice, published in the journal Cell. O’Toole became suspicious of the research after she was unable to reproduce a key experiment conducted by Imanishi-Kari and found discrepancies between the data recorded in Imanishi-Kari’s laboratory notebooks and the data reported in the paper. The case made national headlines, in part, because Nobel Prize-winning molecular biologist David Baltimore was one of the coauthors on the paper, even though he was never implicated in the scandal. A Congressional committee headed by Rep. John Dingell discussed the case during its hearings on fraud in federally-funded research. In 1994, the Office of Research Integrity, which oversees NIH-funded research, found that Imanishi-Kari committed misconduct, but a federal appeals panel overturned this ruling in 1996 (Shamoo and Resnik 2015).

In March 1989, University of Utah chemistry professor Stanley Pons and Southampton University chemistry professor Martin Fleischmann announced at a press conference that they had developed a method for producing nuclear fusion at room temperatures (i.e. “cold fusion”). Pons and Fleischmann bypassed the peer review process and reported their results directly to the public in order to protect their claims to priority and intellectual property. When physicists and chemists around the world tried, unsuccessfully, to reproduce these exciting results, many of them accused Pons and Fleischman of conducting research that was sloppy, careless, or fraudulent. While it does not appear that Pons and Fleischmann fabricated or falsified data, one of the ethical problems with their work was that they did not provide enough detail in their press release to enable scientists to reproduce their experiments. By leading their colleagues on a wild goose chase, they wasted the scientific community’s time and resources and tainted cold fusion research for years to come (Shamoo and Resnik 2015).

In March 2014, Haruko Obokata, a biochemist at the RIKEN Center for Developmental Biology in Kobe, Japan, and coauthors published two high-profile papers in Nature describing a method for converting adult spleen cells in mice into pluripotent stem cells by means of chemical stimulation and physical stress. Several weeks after the papers were published, researchers at the RIKEN Center were unable to reproduce the results and they accused Obokata, who was the lead author on the papers, of misconduct. The journal retracted both papers in July after an investigation by the RIKEN center found that Obokata had fabricated and falsified data. Later that year, Obokata’s advisor, Yoshiki Sasai, committed suicide by hanging himself (Cyranoski 2014).

When irreproducibility does not result from misconduct, it can still have serious consequences for science and society. Irreproducible results could cause severe harms in medicine, public health, engineering, aviation, and other fields in which practitioners or regulators rely on published research to make decisions affecting public safety and well-being (Horton 2015). Irreproducible research, which does not have any immediate practical applications, may still have a negative impact on science by causing researchers to rely on invalid data. Researchers need to be able to trust that published data are reliable, and reproducibility problems can undermine that trust (Shamoo and Resnik 2015). Moreover, the public needs to be able to have confidence in the reliability and integrity of science, and irreproducible research can undermine that trust.

Pressures to produce results and publish may be important factors in science’s reproducibility problems (Horton 2015, Shamoo and Resnik 2015). Researchers who are trying to publish results to advance their careers or meet deadlines imposed by supervisors or sponsors may cut corners when designing and implementing experiments. For example, if a researcher has obtained a result that is marginally statistically significant, he or she may decide to go ahead and publish the result without replicating the result and carefully considering whether it is a false positive.

The growing number of for-profit, open access scientific journals which charge high publication fees, i.e. “predatory journals,” may also exacerbate reproducibility problems (Clark and Smith 2015). These journals often promise rapid publication and have negligible peer review. While it is not known how many articles published in these journals report irreproducible results, the poor peer review standards found in these journals present a significant threat to the quality and integrity of published research (Beall 2016).

Adherence to some commonly recognized principles of responsible conduct of research (RCR) plays an important role in promoting reproducibility in science. One of the key pillars of RCR is that scientific records, including laboratory notebooks, protocols, and other documents, should describe one’s research in sufficient detail to allow others to reproduce it (Schreier et al 2006, Shamoo and Resnik 2015). Records should be accurate, thorough, clear, backed-up, signed, and dated. Failure to record a vital piece of information, such as a change in an experimental design, the pH of a solution, or the type of food fed to an animal, the time of year, can lead to problems with reproducibility (Buck 2015, National Institutes of Health 2016, Firestein 2016).

However, it is important to note that scientists may not always know all the factors that could impact research outcomes. For example, Sorge et al (2014) found that exposure to human male odors, but not female odors, induces pain inhibition and stress in mice and rats. Failure to record the sexes of the researchers conducting experiments on rodents involving measurements of pain responses could therefore undermine reproducibility. Prior to the publication of this finding, many researchers would have not recorded or reported the sexes of the experimenters, or taken this information into account in studies or pain or stress in rodents. If researchers are having difficulty reproducing the results of an experiment, they may need to reexamine methods, materials, and procedures to determine whether they have overlooked an important detail.

Transparency involves the honest and open disclosure of all information related to one’s research when submitting it for publication (Landis et al 2012). Information may be disclosed in the materials and methods section of the paper or in appendices. Because many journals have space constraints that limit the length of articles published in print, some disclosures may need to occur online in supporting documents. Many journals require authors to make supporting data available on public websites (Shamoo and Resnik 2015).

While required disclosures depend on the nature of research one is conducting, some general types of information which should be disclosed include: the research design (e.g. controlled trial, prospective cohort study), methods (e.g. blinding, randomization), procedures, techniques, materials, equipment, data analysis methods and tools (including computer programs or codes), study population (for animals or humans), exclusion and inclusion criteria (for animals and humans), ethics committee approvals (if appropriate), theoretical assumptions and potential biases, sources of funding, and conflicts of interest (Landis et al 2012, Nature 2014b, McNutt 2014, Nature 2015, Elliott and Resnik 2015, Rooney et al 2016, Morgan et al 2016). Additional disclosures may need to occur after the research is published to allow independent scientists to obtain information needed to reproduce experiments, reanalyze data, or develop new hypotheses or theories related to the research.

Some researchers may decide to retract a paper or publish an expression of concern if its results cannot be reproduced. For example, Casedevall et al (2014) examined 423 retraction notices indexed in PubMed that cited error (as opposed to misconduct) as the reason for the for retraction and found that 16.1% of these were retracted due to irreproducibility. Other types of errors included laboratory error (55.8%), analytical error (18.9%). 9.2% of the retraction notices were classified as “other” error.

The Committee on Publication Ethics (2009) has developed guidelines for retracting articles. According to the guidelines, “Journal editors should consider retracting a publication if: they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error); the findings have previously been published elsewhere without proper crossreferencing, permission or justification (i.e. cases of redundant publication); it constitutes plagiarism; or it reports unethical research (Committee on Publication Ethics 2009: 1).” The purpose of retracting an article is to provide a “mechanism for correcting the literature and alerting readers to publications that contain such seriously flawed or erroneous data that their findings and conclusions cannot be relied upon (Committee on Publication Ethics 2009: 2).” Retraction notices should provide a clear explanation of the reasons for the retraction and should be linked to the original article. Retracted articles should be clearly identified in electronic versions of the journal and bibliographic databases (Committee on Publication Ethics 2009: 2).”

To help students and trainees better understand how to promote reproducibility in their work, courses in research methodology and RCR should include sections devoted to the importance of reproducibility and issues that can impact it, such as experimental design, record-keeping, biological variability, data analysis, and transparency (Titus et al 2008, Shamoo and Resnik 2015, National Institutes of Health 2016). The NIH has developed some online reproducibility training modules which are available to the public (National Institutes of Health 2015). The modules address topics such as blinding, randomization, transparency, record-keeping, bias, sample size, and data analysis. Several World Conferences on Research Integrity (2016) have provided forums for international discussion of ethics issues in research and science education, including those that impact reproducibility.

Researchers should also informally discuss reproducibility issues with their students and trainees as part of the mentoring process. For example, if a student or trainee is having difficulty repeating an experiment, the mentor should help him or her to understand what may have gone wrong and how to fix the problem. The mentor may also be able to share stories of his or her own experimental successes and failures with the student to illustrate reproducibility concepts. Failure to replicate an experiment need not be a total loss but can be an opportunity to teach students about principles of good science (Firestein 2016).