Editorial: Reproducibility in Research: A Cautionary Tale and Lessons Not Learned

The 1970s and 1980s were a transformative period for neuroendocrine research. During these 2 decades, many of the hypothalamic peptides regulating pituitary hormone secretion were identified and the field exploded with studies characterizing the spectrum of their biological activities. In the race to these exciting discoveries, discordant results were published regarding the central actions of the newly discovered GH-releasing factor (GRF). Two groups reported a biphasic effect of central administration of human GRF with low doses suppressing and high doses stimulating GH release in rats (1, 2). In contrast, a third group reported that administration of both low and high doses of human GRF inhibited GH secretion (3). Through meticulous scientific detective work, the source of the discrepancy was uncovered and, in the best scientific tradition, promptly published (4). Chemical characterization of the peptide that produced GH inhibition at all doses demonstrated that the putative GRF peptide consisted predominantly of corticotropin-releasing factor. Furthermore, use of new GRF samples reproduced the previously reported stimulatory effect on GH. Although how the original sample became contaminated is not known (4, 5), the source of this error is irrelevant for the purpose of scientific progress: the truth was discovered and revealed.

This episode provides a beautiful example of the self-correction that is an essential part of the scientific method. It is what happens when investigators are committed to finding and reporting the cause of discrepant results and journals publish the new findings. In fact, the following admonition occurs in the report that identified the problem (4): “We would recommend that identification of lot numbers for peptides be provided in publications and that a small quantity be retained for subsequent analysis in order to speed resolution of problems of this nature in the future.”

Contrast this historical event with what often happens today when academic laboratories obtain proprietary compounds or reagents with no information about their structure or composition. The sole source for such investigational materials is the group that developed them, usually a biopharmaceutical company or specialty reagent supplier. In the best case, the structures and/or composition of the materials provided are published. However, this does not generally occur. Many papers in the Endocrine Society and other journals describe results with what I will here call “magic powders.” These reagents are usually identified by an alphanumeric code that is useful only to the group supplying the material. The code provides no information either to the recipient or to the scientific community. How is it that many scientists and journals consider such ignorance acceptable and unavoidable?

Compounds that have the potential for therapeutic use are of enormous interest to both basic and clinical endocrinologists. Academic scientists are excited to work with pharmaceutical and biotechnology companies to learn about the activity of new compounds designed to act on their favorite signaling target and companies are delighted to have the results of their studies. The willingness of companies to share drugs in development and, even more, to provide monetary support for research is greatly appreciated by academic scientists, especially in the current fiscally constrained environment.

In fact, collaboration between pharmaceutical companies and academic investigators provide many advantages to both. The former gain the opportunity to determine the activity of new compounds in a much broader array of biological systems than available in their own laboratories and the latter gain access to potential new drugs to evaluate in their favorite system, thus providing exciting opportunities for discovery. Importantly, such collaborations decrease the cost of scientific and medical research both in dollars and in time by taking advantage of the specialized expertise of each group of investigators. Because of these advantages, academic investigators and universities have been willing to sign Material Transfer Agreements, Sponsored Research Agreements, and Service Contracts in which they agree not to determine the structure, composition, or purity of the materials provided for testing. An example of the language in such agreements is as follows: “[We agree to] keep absolutely secret and confidential, and not disclose or permit to be disclosed [any information from the supplier and] not to analyze, reverse engineer or otherwise attempt to identify the materials [provided].” These restrictions often extend for many years beyond the term of the original agreement. It is understandable that companies' proprietary information must be protected in order to safeguard the considerable investment that goes into developing potential new drugs. However, without information about the structure and composition of materials used in research, the benefits of the acquired knowledge for the scientific community is severely limited and independent reproduction of results is impossible. Moreover, contracts or agreements that prohibit analysis of experimental materials and reagents prevent investigators from troubleshooting discrepant results. Publication of studies with magic powders does a disservice to the scientific enterprise and, as journals, we have a responsibility to ensure that published reports provide sufficient information to reproduce, fully analyze, and build on reported experiments.

Magic Powders. (Adapted from Shutterstock Image ID 225155713, © warlord76/Shutterstock.com)

In 2003, the Publications Committee of the Endocrine Society discussed appropriate policies pertaining to the chemical identity of research compounds described in submitted manuscripts and approved this statement to be incorporated into the Instructions to Authors of all Endocrine Journals. “Manuscripts describing experiments with new compounds must provide their chemical structures. For known compounds, the source or literature reference to the chemical structure and characterization must be provided.” Unfortunately, enforcement of this policy has been poor. In keeping with the letter, but not the intent of the policy, authors may reference patents that contain the structure of the material in magic powders. However, because these patents usually claim protection for many compounds, the structure of the particular reagent(s) used in a journal report remains impossible to determine.

In the last few years, much attention has been focused on the resources wasted by the inability of laboratories to reproduce published research and on the damage produced by irreproducible research to the integrity and credibility of the entire scientific enterprise (6–10). An important contributor to difficulties in reproducing published results is believed to be the minimal experimental detail provided in published scientific papers (11). This belief has fueled the creation of new journal guidelines to increase the detailed methodological information to be included in submitted manuscripts (12–14). New recommendations state that submitted manuscripts should uniquely identify critical reagents and materials used. For example, detailed identification and validation of antibodies and mammalian cell lines is specifically recommended or required (15–17). However, description of the structure and composition of crucial chemical reagents has not garnered much attention. Reproducibility, validation, and extension of published results constitutes the essence of scientific progress, and these depend absolutely on the disclosure of the structure and composition of reagents. In December 2014, the Endocrine Society submitted this recommendation to the National Institutes of Health regarding Reagent-Related Barriers to Reproducible Research: “Investigators should both know and report the specific nature of all reagents.” Clearly, all participants in the scientific research enterprise must act to ensure that investigators know and disclose the chemical nature of the materials used in published experiments and that they have the right to analyze these materials as necessary to guarantee the validity of their results.

Agnes Schonbrunn, PhD